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Executive summary 


The project 


The Parent Engagement Project (PEP) was a school-level intervention designed to improve pupil 
outcomes by engaging parents in their children’s learning. The intervention involved text messages 
being sent to parents using school communications systems, such as Schoolcomms. Texts informed 
parents about dates of upcoming tests, whether homework was submitted on time, and what their 
children were learning at school. The programme was developed collaboratively by research teams from 
the University of Bristol and Harvard University and was delivered between September 2014 and July 
2015. The trial involved 15,697 students in Years 7, 9 and 11 from 36 English secondary schools, with 
schools sending an average of 30 texts to each parent over the period of the trial. 


This study was an efficacy trial in which the developers of the intervention managed its delivery to ensure 
optimal implementation. It was a cluster randomised controlled trial with randomisation at the Key Stage 
level, designed to determine the impact of the intervention on the academic outcomes of students in 
English, maths, and science, and the impact on absenteeism. A process evaluation used focus groups, 
telephone surveys, interviews, and an online survey to provide data on implementation and to capture 
the perceptions and experiences of participating parents, pupils, and teachers. The project was co- 
funded by the Education Endowment Foundation (EEF) and Nominet Trust as part of a funding round 
focused on digital technology. 


Key conclusions 


. Children who had the intervention experienced about one month of additional progress in maths 
compared to other children. This positive result is unlikely to have occurred by chance. 


. Children who had the intervention had reduced absenteeism compared to other children. This 
positive result is unlikely to have occurred by chance. 


. Children who had the intervention appeared to experience about one month of additional progress 
in English compared to other children. However, analysis suggests that this finding might have 
been affected by bias introduced by missing data, so we cannot reliably draw this conclusion. 
There is no evidence to suggest that the intervention had an impact on science attainment. 


. Schools embraced the programme and liked its immediacy and low cost. Many respondents felt 
that the presence of a dedicated coordinator would be valuable to monitor the accuracy and 
frequency of texts. Schools should consider whether they would be able to provide this additional 
resource. 


. The vast majority of parents were accepting of the programme, including the content, frequency, 
and timing of texts. 


How secure are the findings? SECUMy falng aWareed a\part 
of the EEF peer review process 


Overall, the findings have moderate security. The trial was a two-armed multi-site cluster randomised 
controlled trial. The trial was well-designed and large. However, pupils from 19% of the schools were 
not included in the analysis because the schools dropped out of the trial. There were some important 
differences between the pupils who received the intervention and those in the comparison group; the 
latter had a higher proportion of students eligible for free school meals. 


Boycotts of the SATs tests by schools in 2009/2010 meant that pre-intervention attainment data was 
missing for a relatively high proportion of pupils who could not therefore be included in the analysis. 
Tests were conducted to assess the impact of this missing data on the results. These tests supported 
the findings that the programme had a small positive impact on attainment in mathematics and 
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absenteeism, but suggested that the result for English is less certain and might be the result of bias 
introduced by missing data. 


What are the findings? 


There were small positive impacts on maths and English, and a reduction in absenteeism. As discussed 
above, the evaluation team conducted tests to check that missing information did not bias the results. 
These tests supported the impact on maths and absenteeism but found a risk of bias to the English 
outcome. The positive effects of the programme in relation to a reduction in absenteeism was surprising 
given that no texts related to attendance were sent, and is perhaps related to the increased monitoring 
by parents of children’s school-related activities overall, creating an environment in which pupils felt less 
able or willing to truant. It appears the intervention was more effective in reducing absenteeism in the 
KS4 cohort compared to KS3. This result is of particular note given that absenteeism increases as pupils 
progress through school. 


Of the parent outcomes that were measured, the only positive effect that was unlikely to have occurred 
by chance was in relation to the likelinood of parents asking their child about revising for upcoming tests, 
where intervention parents were more likely to do so than control parents. 


Although the effects of the intervention were low, the programme is inexpensive and relatively easy for 
schools to implement. Parents reported general satisfaction with the frequency, content, and timing of 
the texts, and in the majority of cases talked to their children about the information they were receiving 
via text from the school. Similarly, the teacher survey and interviews showed that schools were 
enthusiastic about PEP and liked its immediacy and low cost. Teachers acknowledged that there would 
need to be a person (or persons) responsible for the coordination of texts within the school—to monitor 
text content, accuracy, and frequency. Some schools struggled with the more complex and time- 
consuming texts, such as conversational prompts, and sent these less frequently. 


How much does it cost? 


The intervention cost a total of £7,550 per school (£7.55 per pupil) for the first year of implementation, 
including training and cover for the staff involved. The annual cost for subsequent years would be £3.25 
per pupil per school to cover the cost of 65 texts. Schools that do not already have access to suitable 
communication software would need additional resources for this. The price of the software varies, but 
for Schoolcomms is £1,500 a year, bringing the average cost per student over three years to around 
£5.68 a year. 


Effect size Estimated 


Outcome Loe) my ot) oy Ey (95% confidence months’ SAU 
: rating 
interval) progress 
: 0.033 
English : 
os Sere (0.004, 0.064) ue 
0.067 
Math £ 
aths 5,613 (0.041, 0.092) +1 84a 
: -0.013 
£ 
Science 4,726 (-0.048, 0.022) 0 84a 
0.054 Absenteeism 
Absenteeism 7,436 (-0.079, -0.029) reduction of aaa £ 
one half day 


1 Since this report was published, the conversion from effect size into months of additional progress has been slightly revised. If 
this result was reported using the new conversion, it would be reported as 0 months of additional progress rather than +1. See 
here for more details. 
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Introduction 


This report presents the findings from an efficacy trial and process evaluation of the Parent Engagement 
Programme (PEP). The study was conducted by the Centre for Effective Education, Queen’s University 
Belfast between February 2014 and February 2016. 


Intervention 


PEP is a universal, school based intervention that is designed to help secondary schools engage 
parents in their child’s learning, with the ultimate aim of improving pupil attainment. It is intended to be 
a low-cost and low-effort process, involving text messages being sent directly by schools—via their 
existing school information management system (‘SIMS’, Schoolcomms or similar system)—to parents’ 
mobile phones at specific time-points and intervals during the school year. Parents were not made aware 
of the specific type of text content to expect, but rather that they would receive text messages about 
their child’s progress in school. The programme was developed collaboratively by a research team from 
the University of Bristol and Harvard University. 


For the purpose of this evaluation, the programme was delivered from September 2014 to July 2015 
and focused on the subjects of English, mathematics and science. The type of text messages that 
parents were sent included some or all of the following information (more detail is provided in Table 1): 


e advance notice of any upcoming tests; 

e notice of missing homework; and 

e brief updates on what their child learned in their lessons, with conversational prompts (these 
subject texts were sent on a three-weekly rotation—science one week, maths the next, and 
English the following week, then the fourth week it was back to science). 


While the intervention is suitable for all pupils in a secondary school setting, the recipients of the 
intervention in this study were Year 7, 9 (Key Stage 3), and 11 (Key Stage 4) pupils. It was intended 
that parents would receive approximately 65 texts over the academic year. For example, per subject, 
ten texts would be sent regarding upcoming tests and conversation prompts were to be sent once per 
week during non-testing weeks (totalling five texts over the academic year). It was also planned that 
parents would receive five texts per subject for feedback on their child’s grades or test results, and five 
texts concerning their child’s attendance at school, but these texts were not sent as part of the project. 
There were no targets or standards set for the frequency of missing homework text messages as this is 
dependent on whether homework is submitted or not. 


Table 1: Description of the text messages that were sent as part of the PEP intervention 


a> qmay/ oy: Description Requirement from school Text sent by 


4 and 1 day warnings of Share assessment calendar 
upcoming tests and important | with project delivery team, at 
deadlines least 1 week in advance 


Upcoming 
assessment 


Project delivery 
team 


Missing Notification of missing 


hamiawork Ramewane Log behaviour report on SIMS Automatic 


Conversation | Summary of the day's lesson | Teachers must write prompts | Project delivery 
prompt so parents can discuss as timetabled team 
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Training and support 


The intervention is designed to be implemented by teachers and administrators within the school. For 
the purpose of this trial, however, schools were supported in the implementation of the programme by 
the project delivery team. 


Teachers were required to attend 1.5 days training—one full day in September 2014 and a further half- 
day training event the following January. At these events schools were introduced to key members of 
the project delivery and evaluation teams as well as provided with information packs for their teachers 
and administrators. Training was digitally recorded and made available to all schools. 


The training events were hosted by the project delivery team and attendees were provided with an 
overview of the study, its design, and the associated timelines. Schools were introduced to an updated 
version of Schoolcomms specifically developed for schools taking part in the study. Download and set- 
up instructions were provided by way of written information and a practical demonstration. Attendees 
were also provided with detailed written and verbal instructions on how to implement the five 
components of the programme: texting (1) advance notification of upcoming assessments (ideally 
approximately ten test texts per subject per year), (2) the results of tests (n=5), (3) missing homework 
alerts (as appropriate), (4) conversational prompts regarding recent learning (n=5), and (5) attendance 
summaries (n=5). 


With respect to modifying the intervention, schools in the current study were required to adhere to the 
protocol described above and were only allowed to modify text content related to the conversational 
prompts. It was expected that there would be some variability between schools in relation to the 
frequency of texts sent, and this was indeed the case (described in more detail later in the report). 


In each school a member of staff was appointed to act as the PEP project liaison officer. Their role was 
to co-ordinate the delivery of PEP within the school and ensure that the appropriate content for the texts 
was collated from individual subject teachers. This content was subsequently given to the project 
delivery team who took responsibility for sending the information to parents via a text from the 
Schoolcomms system. 


Within schools, use of texting systems is often restricted to administrators and is unfamiliar to teachers. 
For this reason—and to ensure consistency of implementation—the project delivery team employed a 
number of research assistants (RAs), each of whom had dedicated responsibility for overseeing and 
supporting the implementation of PEP in a certain number of the evaluation schools (usually eight). 
Each RA would liaise directly with the PEP project liaison officer who would in turn liaise with the 
teachers and staff involved in the project within the school. 


The specific role of the RA included: 


obtaining test dates from heads of department; 


obtaining conversation prompts from heads of department; 


encouraging the use of the missing homework texting tool; and 


providing any ongoing support needed by the schools. 


To facilitate the smooth running of the project, RAs ensured that they introduced themselves to the 
heads of department and built personal relationships with school staff. They were encouraged by the 
delivery project lead to take the time to understand the individual circumstances of each school and 
when test dates were likely to be available. RAS were supported by the project lead who held daily and 
weekly Skype meetings to allow each RA the opportunity to raise issues and to get advice on how to 
deal with any problems arising. 
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In the future, if the programme is rolled out more widely, it is envisaged that schools would assume 
responsibility for sending the texts as most schools already possess the necessary technological 
knowledge and resources. In this study, because it was an efficacy trial, the delivery team managed the 
sending of the text messages to ensure the programme was delivered under optimal conditions. Despite 
this, the consistency of implementation did vary considerably between schools and this is described in 
more detail in the fidelity section of the process evaluation. 


Background evidence 


Parental engagement plays a significant role in children’s learning (Goodall and Vorhaus, 2010) and 
can be thought of as ‘parent engagement in their children’s lives to influence the child’s overall actions’ 
(Kim, 2009, p. 89). While the most powerful form of parental engagement takes place between parents 
and their children entirely away from school (Sylva, Melhuish, Sammons et al., 2008), the school has 
much to gain by facilitating and promoting such engagement. ‘School parental engagement’ is 
considered to involve a parent’s participation in their child’s learning as well as parental involvement 
with schools (Harris and Goodall, 2007). Thus it brings together two key contexts, the home and school 
settings, within the educational pyramid of pupil-school-home. 


Parental engagement with children’s learning has recently emerged as one of the best levers available 
for raising achievement (Gorard, See and Davies, 2012)—by as much as 0.44 of a standard deviation 
(Fan and Chen, 2001; Jeynes, 2007). The benefits of both home-based and school-based parental 
involvement, in terms of improving academic achievement (for children of all ages), have been reported 
in several meta-analyses and reviews of the literature (Henderson and Mapp, 2002; Cox, 2005; 
Pomerantz, Moorman and Litwack, 2007; Fan and Chen, 2001; Jeynes, 2007). It has been identified, 
albeit by correlational studies, as a key factor associated with improved behaviour, increased 
attendance at school, better emotional adjustment, and greater well-being (Epstein, 2001; Henderson 
and Mapp, 2002; Jeynes, 2007; El Nokal et a/., 2010, Hornby and Lafaele, 2011). 


While there is evidence to suggest that school-based parental engagement can result in raising 
achievement, the evidence relating to the long-term impact of such programmes on academic outcomes 
is limited and typically studies have not employed rigorous methodological designs. Consequently—and 
despite the established correlation between parental engagement and positive outcomes for students— 
it is still necessary to understand and identify what parents want and need in order to help them to better 
engage and participate in their children’s learning experiences. This remains a challenge for schools, 
particularly in relation to reaching and involving parents who are unwilling or unable to engage with their 
child’s learning (Goodall and Vorhaus 2011; See and Gorard, 2014). Traditional engagement strategies 
used by schools, such as talking with parents on the phone, parent meetings, and school visits, are 
often ineffective and do not truly support parent involvement (Wyn et al., 2000). Indeed, Fryer (2016) 
examined evidence from experimental studies in which the treatment involved providing parents with 
information about their child’s education, and concluded that parent information interventions tended to 
have no impact on standardised test scores for school-aged children. Moreover, it appears that the 
engagement young people value the most is the interest of their parents—parents asking about school 
days and offering guidance, as opposed to attending parents evenings and responding to school 
correspondence (Harris and Goodall, 2008)—and, as the literature suggests, it is this type of 
engagement that can have a greater effect on achievement: a meta-analysis of 51 studies (Jeynes, 
2012) found that effect sizes for school-based parental involvement programmes were typically not as 
large as those found for voluntary acts of parental involvement (ES = 0.30 compared to ES = 0.51). 


Recent advances in information communications technology (ICT) have opened up new ways and 
opportunities for schools to communicate with parents. ICT provides a convenient means for schools to 
send, and parents to access, up-to-date information about their child’s learning (Goodall and Vorhaus, 
2011) and the curriculum (Lewin, Mavers and Somekh, 2003). Research shows that providing additional 
information to parents can have a significant impact on children’s attainment. For example, Bergman 
(2014) found that notifying parents of missing homework reduced any further replication by 25%. In the 
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U.S., Kraft and Rogers (2015) found that informing parents via weekly one-sentence messages that 
contained feedback about student exam performance reduced the proportion of students failing to earn 
summer school course credit by 40%. Moreover, York and Loeb’s (2014) evaluation of a pre-school text 
messaging programme which aimed to help parents (n = 440) support their children’s literacy 
development concludes that—given its widespread use, cost effectiveness, and ease of adaptability— 
regular text message prompts and guidance (three texts per week, over an eight-month period) to 
encourage parents to practice literacy with their children led to improvements in children’s attainment— 
by as much as 0.34 of a standard deviation. There is no consensus within the literature as to whether 
this type of approach is moderated by different characteristics of students or schools, so with this in 
mind the current study sought to explore whether the efficacy of the intervention might be affected by 
student or school characteristics. 


According to the Department for Education (Goodall and Vorhaus, 2011), a top government priority is 
the development of effective school-based programmes for supporting parental engagement. This report 
determines the effectiveness of various intervention components in supporting parental engagement 
and improving outcomes for children by profiling what success looks like. In particular, based on a review 
confined to studies undertaken between 2000 and 2010, the report presents a model of good practice 
for promoting parental engagement within an educational setting. The model suggests four key 
elements, namely: 


e planning—that parental engagement strategies must be designed as a whole-school 
approach, informed by a comprehensive needs assessment; 


e leadership—the need for a senior teacher who has a strong sense that children and families 
matter, is focused on people and outcomes, is willing to develop relationships and networks, 
functions as an effective role model, and engages support from stakeholders; 


e collaboration—engagement with all parents that is proactive rather than reactive, empowering, 
sensitive, and cognisant of the contributions parents can offer; and 


e sustained improvement—ongoing support, monitoring, development, and evaluation. 
Rationale for the current study 


With the above in mind, the current intervention, PEP, seeks to promote the engagement of parents in 
their child’s learning by building on communication processes and technologies that are already familiar 
to schools. Here, through the creative use of text messages from the school, parents are kept informed 
about various aspects of their child’s attendance, learning, and progress. 


The project delivery team has already undertaken developmental work to better understand the type 
and frequency of text messages that are likely to be most effective in improving attainment outcomes. 
In a feasibility study it was found that pre-informing parents about upcoming tests emerged as relatively 
straightforward and more effective compared to texting conversation prompts. The latter, it was 
concluded, should be limited to non-assessment weeks and rotated between departments to avoid over 
burdening parents. It was also concluded that missing homework reports could be automated and 
attendance reports sent at the school’s discretion. The findings from this study (Chande, et al., 
forthcoming) have subsequently informed the version of PEP that is being trialled in the current 
evaluation. As such, this is an efficacy trial that aims to provide an initial test of the programme's efficacy 
under optimal conditions (that is, delivered by schools with support from the project delivery team). It 
represents an opportunity to conduct a rigorous experimental evaluation that will enable us to better 
understand the causal mechanism between parental engagement (as defined by PEP) and pupil 
attainment. Furthermore, as the data from this trial will be linked to the National Pupil Database (NPD), 
and potentially to other relevant statutory datasets in the future, there is the potential to follow up 
participants to better understand the long term effects, if any. 
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The project and evaluation was funded as part of the EEF’s Digital Technology in Education funding 
round (2013) in partnership with Nominet Trust. The aim of this funding stream was to identify and 
evaluate promising interventions that used digital technology to improve educational outcomes, 
specifically for pupils from disadvantaged backgrounds. 


Evaluation objectives 


The purpose of the current evaluation was to determine the impact of PEP on the academic outcomes 
of secondary school students and explore issues related to implementation and delivery. The evaluation 
consisted of two components: 


An impact evaluation that aimed to— 


e assess the impact of the intervention on the English, maths, and science outcomes (primary 
outcomes) in addition to school attendance of pupils, and the engagement behaviour of 
parents (Secondary outcomes); and 


e determine the impact of the intervention for different groups of students, specifically, whether 
the impact of the programme varied according to Key Stage group, gender, ethnicity, 
socioeconomic/FSM status, school size, or baseline attainment. 


A process evaluation that aimed to explore whether the intervention was delivered with fidelity, and to 
identify the successes and challenges associated with implementation. 


Project team 


PEP is a collaborative research project and the intervention was developed by Professor Simon 
Burgess, Raj Chande (University of Bristol) and Assistant Professor Todd Rogers (Harvard University). 
Raj Chande led the delivery and implementation of PEP in the evaluation schools with the support of 
James Watson, John Ault, Erica Brandt and Ben Cole (University of Bristol), and Kim Bohling (Harvard 
University). The project delivery team at the University of Bristol provided technical support and 
assistance directly to schools. 


Dr Sarah Miller (Queens University Belfast) led the independent evaluation team, which included Dr 
Jenny Davison, Dr Jamie Yohanis, Dr Seaneen Sloan, Aideen Gildea and Professor Allen Thurston. 
The evaluation team was responsible for the design of the evaluation and all data collection for both the 
impact and the process evaluation. 


Ethical review 


Ethical approval was granted by the Research and Ethics Committee of the School of Education, 
Queens University Belfast in April 2014. Informed consent was obtained at the school level from 
headteachers and teachers. Written information and opt-out consent forms were sent home to parents 
of eligible pupils. Opt-in consent was obtained for pupils taking part in the focus groups. 


Trial registration 


This trial is registered with the ISRCTN registry (ref: ISRCTN1696856). 
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Methods 


Trial design 


The evaluation was designed as a multi-site cluster randomised controlled trial (CRCT) to determine the 
impact of the intervention on the academic outcomes of secondary school students. The study involved 
two Key Stage groups, KS3 and KS4, and three year groups—Years 7 and 9 from KS3, and Year 11 
from KS4—in a convenience sample of 36 geographically dispersed English secondary schools. 


Within-school random assignment was used to allocate Key Stage group (KS3 and KSA) to either 
intervention or control. Consequently all participating schools were involved in the delivery of the 
programme. Pupils and parents in the intervention Key Stage received the Parental Engagement 
Programme for one academic year (September 2014—July 2015). Those in the control Key Stage did 
not receive the intervention and represented ‘business as usual’. 


Key Stage group was chosen as the unit of randomisation for two main reasons. Firstly, it maximised 
the efficiency of the trial, requiring fewer schools than if ‘school’ were the unit of randomisation. Secondly 
it minimised the risk of contamination between treatment and control arms compared to a simple 
randomised trial using ‘pupil’ as the unit of randomisation. 


Protocol violations 


There were two changes to the protocol, both of which were related to outcomes, specifically science 
attainment and parent behaviour and attitudes. 


Firstly, an amendment to the protocol (dated 23 February 2015) indicated that science attainment in the 
Year 7 cohort of participants would not be measured due to the lack of an available and suitable test. 
There was, however, a protocol violation of this updated protocol because, as it transpired, the 
evaluation team did measure this outcome for Year 7 using an adapted version of the 2009 Standard 
Assessment Test for Science (see the following section on outcome measures for more detail). 


Secondly, it was originally intended that the evaluation team would collect secondary outcome data 
relating to parental behaviour and attitudes at post-test via a self-complete questionnaire. This data was 
collected, but not by the evaluation team. Instead, the project delivery team had the opportunity and 
resources to complete a much more robust and in-depth study on parental behaviour and attitudes than 
was possible within the remit of the current evaluation, so they took forward this aspect of the study. 
This data was provided by the project delivery team, analysed by the evaluation team, and is reported 
below. 


Participant selection 


Between January and April 2014 the project delivery team recruited secondary schools from across 
England drawing on established relationships with networks of schools across the country. While this 
method of recruitment worked well in terms of achieving the required sample size, the lack of random 
sampling is a threat to the external validity of the trial and means that the generalizability of the results 
may be somewhat compromised. 


Schools were eligible for inclusion in the evaluation if they were willing to be randomly assigned to 
condition at the level of the Key Stage group (KS3 or KS4), engage with the intervention and implement 
it with students and parents, provide access to socio-economic and demographic data of students, and 
provide access to assessment data for participating students. Within participating schools, all pupils in 
Years 7, 9 and 11 were eligible for inclusion. 
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School-level consent was sought from headteachers prior to randomisation. A Memorandum of 
Understanding (MoU) that detailed schools’ responsibilities and rights regarding the project was signed 
by all participating headteachers (Appendix 2). Schools informed the parents of all Year 7, 9 and 11 
pupils about the evaluation using the information and consent forms provided by the evaluation team 
(Appendix 3). Parents had the opportunity to withdraw their child from the evaluation by returning the 
opt-out consent form to the relevant class teacher, and 285 chose to do so. 


In September 2014, an information and training event was held for two representatives from each of the 
participating schools. This was organised by the University of Bristol project delivery team and was also 
attended by members of the evaluation team. The event provided a valuable opportunity to explain both 
the programme and the nature of the independent evaluation to teachers. 


Outcome measures 
Primary outcomes—student achievement in English, maths and science 


The intervention was targeted at English, maths and science classes, thus the co-primary outcomes 
were student attainment in these three subjects at immediate post-test, at the end of the school year 
(May 2015). All participating students were tested on all outcomes. Post-test data collection was 
organised and overseen by the evaluation team—the outcome measures are described below and in 
Table 2. To control for prior attainment, KS2 data on English and maths outcomes for all participating 
pupils was accessed by the evaluation team via the NPD. 


Table 2: Primary outcomes and post-test measures for each Key Stage and year group 


CAVES) 10 [2] Year Group Primary outcomes and measures 


Access Reading | Access Maths Test Aue Sat apele 
veel? Test (Hodder) (Hodder) Eee 
Q1-13 
Key Stage 3 
vears Access Reading | Access Maths Test seared me ee 
Test (Hodder) (Hodder) 


Q1-11 
Key Stage 4 GCSE English GCSE Maths GCSE Science 
English and Maths 
Key Stage 3 


For KS3 (Years 7 and 9), attainment in maths was measured using the Hodder Access Maths Test 
(McCarty, 2008) and attainment in English was measured using the Hodder Access Reading Test 
(McCarty and Crumpler, 2014). It was intended that these tests would take the place of the school’s 
normal end of year assessments that would be reported to parents. Pencil and paper versions of the 
tests were administered by the schools so it was not possible to blind teachers administering the tests 
to student allocation. However, the tests were scored independently by Hodder, who returned the test 
results—within three weeks of receiving them—directly to the evaluation team which in turn shared the 
data with schools. 


The Hodder Access Reading and Access Maths tests are appropriate for pupils aged up to 16 and are 
reliable, valid, and widely used measures of literacy and numeracy outcomes. These tests were selected 
because they are more efficient time-wise than other available, comparable tests. 
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Access Reading consists of 60 items that assess four reading skills: literal comprehension, vocabulary, 
comprehension requiring inference, prediction and opinions, and comprehension requiring analysis. The 
test provides a global measure of reading attainment as an age-standardised score and it is this score 
that was used in the analysis. 


Access Maths (Test 2, suitable for Years 7 to 9) consists of 45 items that assess seven numeracy skills: 
using and applying mathematics, counting and understanding number, knowing and using number facts, 
calculating, understanding shape, measuring, and handling data. The test provides a global measure of 
maths attainment as an age-standardised score and it is this score that was used in the analysis. 


Pencil and paper rather than computer based versions of the tests were selected due to this being 
schools’ preferred mode of administration. Computer based tests can add a layer of complexity when 
testing large numbers of students, with schools frequently reporting difficulties in resolving timetabling 
issues associated with computer testing large numbers of students. 


Key Stage 4 


For KS4 (Year 11), English and maths was measured at post-test using GCSE English and maths 
standardised scores (from which the GCSE grade is normally derived). This data was accessed by the 
evaluation team via the NPD for all participating KS4 pupils. The reliability and validity of GCSEs are 
well established and are standardised measures that are high in contextual validity as they constitute 
the main indicators of school and student academic performance. In light of this, it is highly likely that 
teachers in all treatment arms of the trial will be focused on ensuring that students succeed on them. 


Science 


Key Stage 3 


Science has not been tested using Standardised Assessment Tests (SATs) since the KS3 SATs were 
dropped in 2010 and as such no recent or standardised science test exists that is suitable for this age 
group. At the time of this evaluation GL Assessment were in the process of devising a science test 
aimed specifically at KS3 and it was intended that this study might make use of this. Unfortunately the 
test was not available in time so this was not possible. 


In the absence of a suitable alternative it was decided to use past SAT science papers from the year 
2009 for both Year 7 (2009, Paper 2, Levels 3-6, Q1—13) and Year 9 (2009, Paper 2, Levels 5—7, Q1— 
11)? acknowledging that this was going to have potential limitations with respect to the test content being 
relevant to the current syllabus being taught in class. The tests were shortened by omitting the final two 
questions to make it possible to administer within a 45-minute period. 


To ensure that pupils were not already familiar with the content of the tests, the evaluation team asked 
all schools whether they had used past SAT science papers as revision tools with their science classes. 
This, however, was not the case. The tests were administered by teachers under exam conditions and 
were returned to the evaluation team to be scored using the accompanying mark scheme. The 
evaluation team subsequently returned the scores directly to schools. 


Key Stage 4 


2 See sats-papers.co.uk 
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For Year 11, post-intervention science attainment was measured using GCSE Science standardised 
scores (highest point score achieved, as provided by the NPD) accessed by the evaluation team via the 
NPD. 


Secondary outcomes—school attendance and parental behaviour and attitudes 


Attendance data (number of sessions absent) for each student was accessed at post-test via the NPD 
for the year of the evaluation, 2014/2015. Attendance for the previous academic year (for the pupils in 
the trial) was used as a baseline for this outcome. 


It was originally intended to collect data on parent-related outcomes via a short self-complete 
questionnaire at post-test. This was to include background demographic information and data relating 
to aspects of the home learning environment, information regarding parental involvement in their child's 
education and school, and aspirations for their child's education and future employment. To minimise 
the research burden on parents, no baseline data collection was planned. As reported above, this data 
was collected by the project delivery team instead of the evaluation team as the latter had the opportunity 
and resources to complete a much more robust and in-depth study on parental behaviour and attitudes 
than was possible within the remit of the current evaluation. Parent interviews were conducted by 
telephone and the following data was collected in this alternative study on parental behaviour and 
attitudes: 


e Parent demographic data was collected, including relationship to child, ethnicity, education 
level, and primary language spoken at home. 


e Parent engagement was measured using a series of ten questions devised by the project 
delivery team (Q2-11 in Appendix 1) relating to ensuring a child’s attendance at school, 
checking that homework is submitted, ensuring revision for upcoming tests is completed, and 
talking to their child about what they are learning at school. Parents were also asked about their 
beliefs relating to their own influence on their child’s education and beliefs about whether the 
school values parents’ participation in their child’s education. These questions were combined 
into a single variable (parent engagement) which had satisfactory internal consistency 
(Cronbach's alpha = 0.75) and a single factor structure. 


In addition, parents were asked about whether they asked their child what they learned at 
school, whether they talked to their child about their attendance, whether they asked their child 
to revise for an upcoming test, and whether they asked their child about the results of any tests. 


e For intervention parents only, questions were asked concerning the frequency of conversations 
with their child about text messages received from school, the perceived usefulness of texts, 
and the preferred content, frequency, timing, and acceptability of texts. 


This parent outcome data has been linked to the trial data by the evaluation team and is reported below. 
Other student variables 


Pupil background information was collected directly from schools including UPN, gender, date of birth, 
FSM status (free school meal entitlement), English as an Additional Language (EAL) status, and 
ethnicity. 


Sample size 


A previous evaluation of a homework alert messaging system detected an effect size in the order of d = 
0.20 in favour of the intervention group (Bergman, 2012). Thus it was estimated that for the proposed 
trial to detect a minimum effect size of d = 0.20 with approximately 80% power, a total sample size of 
68 Key Stage groups (34 schools) and up to 120 students per KS group (approximately 8,160 students 
in total) would be required. These estimates were calculated using Optimal Design (Version 3.01) and 
are based on a two-level cluster design (where pupil is level 1 and school is level 2) and the following 
assumptions: 
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e Significance level = 0.05 

e Power = 80% 

e §6Effect size = 0.20 

e Estimated intraclass correlation coefficient (ICC) = 0.15 


e Estimated variance shared between KS2 and post-test scores (R?) = 0.5 
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Randomisation 


Once the headteacher of each participating school signed the MoU, the project delivery team sent the 
school’s details to the evaluation team for allocation (36 schools in total). Allocation was undertaken by 
the evaluation team (specifically the Principal Investigator) using the programme Minim.? This uses 
minimisation to create groups that are balanced across pre-specified characteristics likely to be related 
to the outcome under investigation, in this case attainment (Altman and Bland, 2005). For this trial 
minimisation was used to ensure that the intervention and control groups were balanced in terms of the 
proportion of KS3 and KS4 allocated to each condition. This process was also used to ensure that 
schools allocated to deliver PEP to KS3 or KS4 were balanced in relation to school characteristics, 
thereby minimizing the risk of schools delivering the intervention to KS4 being somehow systematically 
different (in terms of attainment, size, or deprivation) to the schools delivering the intervention to KS3. 


Within each school, either the KS4 cohort (19 schools) or the KS3 cohort (17 schools) was allocated to 
the intervention. The unallocated KS cohort within each school formed the control group. 


The evaluation team informed the project delivery team of the outcome of allocation and they in turn 
informed the school. 


Analysis 


Analysis was conducted in Stata version 14 (Stata Corporation, College Station, Texas, USA) on an 
intention-to-treat basis. The initial characteristics of the intervention and control groups were compared 
at baseline in relation to their core characteristics: gender, FSM eligibility, EAL status, SEN status, 
ethnicity, and mean scores on the main outcomes (English, maths, science and absenteeism). 


The main effects of the intervention were estimated using multilevel modelling to take account of the 
clustered nature of the data and a series of models were estimated for each outcome (where pupil is 
level 1 and school is level 2). First, a simple analysis was conducted: the relevant outcome measure at 
post-test forming the dependent variable and the independent variables including a dummy variable 
representing whether the child was a member of the intervention or control group (coded ‘1’ and ‘0’ 
respectively) and pupils’ baseline scores for the outcome variable in question. Then, a series of pupil- 
level and school-level characteristics were added as covariates to control for any baseline differences 
in the variables. Standard errors were then bootstrapped as a test of robustness. Bootstrapping is a 
resampling method that provides an alternative means of estimating the variability within the observed 
data. It is a useful technique to establish the stability of results, for example when differences between 
groups and associated standard errors are small. If the standard error estimates still give rise to small 
p values even after bootstrapping, then this suggests that the statistically significant result is stable. 


The main focus for the analysis was the estimated coefficient associated with the dummy variable that 
represented the difference in mean scores on the respective outcome variable between the intervention 
and control groups, once baseline scores and other covariates were controlled for. This coefficient was 
then used to estimate the effect size of the programme in relation to the respective outcome variable as 
the standardised mean difference between the two groups post-intervention (Hedges’ g). 


Multiple imputation was used to account for missing data and this is described in further detail in the 
results section. 


As different measures were used to measure outcomes for KS3 and KS4, the scores for each measure 
were standardised with a mean of zero and a standard deviation of one prior to combining for the 
purpose of analysis. The mean and standard deviation of this pooled outcome remained zero and one 


3 http://www-users.york.ac.uk/~mb55/quide/minim.htm 


Education Endowment Foundation 16 


Texting Parents 


respectively. The baseline score for each outcome was also standardised. The analysis was based on 
a complete case analysis and missing data was not imputed. Effect sizes were calculated using Hedges’ 
g. The full models containing all the parameter estimates for each model are reported in Appendix 4. 


In addition to the analysis of the main effects, sub-group analyses were conducted to examine any 
differential impact of the intervention for different groups of pupils. Specifically, does the impact of the 
programme vary according to: 


e Key Stage group; 

e gender; 

e baseline attainment or level of absence; 
e school size; or 


e EAL status (added as an additional subgroup analysis in the protocol amendment dated 15 
Dec 2015). 


To estimate the effect of the intervention for children eligible for FSM the main analysis was repeated 
on a subsample of the data relevant to that group. 


Implementation and process evaluation 


A relatively light touch process evaluation ran alongside the cCRCT to provide data on programme 
implementation and to capture the perceptions and experiences of key stakeholders’ engagement with 
the programme. The evaluation team collected all the process-related data and its purpose was 
threefold: 


e toexplore the perceived successes, outcomes, and challenges associated with the intervention 
and its implementation; 


e to determine whether the intervention was implemented with fidelity (with a focus on the content 
and frequency of text messages); and 


e to determine whether and how the intervention might be rolled out more widely to more schools. 


All participating schools were asked by the evaluation team to complete a short online survey in term 
two (June 2015), to capture data relating to the ease of implementation and fidelity to the programme 
schedule. The online survey was completed by 123 school personnel from 29 schools. The sample 
comprised 43 teachers, 38 subject heads, one headteacher, 17 deputy headteachers, six senior 
management team, 13 administrative staff, and five roles were not specified. 


Interviews (both telephone and face-to-face) were conducted with 12 participants from four schools. 
Originally it was intended that three schools would participate in the process evaluation, however due 
to uncertainty around the participation of one school a fourth school was recruited as a reserve. As it 
transpired, all four schools were wiling and able to take part and so they were all retained in this element 
of the study. 


Schools were chosen to represent those that experienced difficulties with implementation as well as 
those who did not, based on both survey responses and school ratings provided by the University of 
Bristol delivery team. Schools were also selected to represent a mix of socioeconomic and ethnic 
backgrounds. Of those interviewed, four respondents were project liaison officers for PEP within their 
school (of which three were teachers and one administrative support staff), seven were teachers, and 
one was a headteacher. 


Four focus groups were conducted with intervention pupils (n = 29), two focus groups from each year 
group within Key Stage 3 (Years 7 and 9). Schools were unable to accommodate focus groups for the 
Key Stage 4 cohort (Year 11) as at the time these pupils were sitting their GCSEs. 
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e Focus Group 1: Year 7 (n = 8, 4 boys and 4 girls) 

e Focus Group 2: Year 9 (n= 8, 4 boys and 4 girls) 

e Focus Group 3: Year 7 (n= 7, 2 boys and 5 girls) 

e Focus Group 4: Year 9 (n = 6, 2 boys and 4 girls) 
The University of Bristol project delivery team kindly provided fidelity data relating to the frequency and 
type of texts sent by each school for the duration of the trial. This is reported below (and in Appendix 5). 


Timeline 


Table 3: Timeline of activities related to the evaluation and intervention delivery 


Date Activity 


Costs 


The project delivery team kindly provided the information on which the cost calculations are based. 
Costs are calculated from the perspective of schools and the costs to the school as well as the cost per 
pupil are reported. These figures include the cost of teacher training in addition to costs associated with 
implementing the programme (both project delivery team support and text message costs). Cumulative 
costs are reported over a period of three years. More detail is provided in the analysis section below. 
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Impact evaluation 


Participants 


Key Stage group was the unit of randomisation, thus within each school either KS3 (Years 7 and 9) or 
KS4 (Year 11) was allocated to receive the intervention. The remaining Key Stage cohort formed the 
control group. In total, 72 Key Stage groups within 36 schools were allocated to the intervention and 
control arms of the trial. 


The intervention group consisted of 19 KS4 groups and 17 KS3 groups (pupil n = 7,570). The control 
group consisted of 17 KS4 groups and 19 KS3 groups (pupil n = 8,127). The total number of pupils 
involved in the trial was 15,697. Participant flow through the study is described in more detail in Figure 
1 below. 


Attrition 


Seven schools withdrew from the trial post allocation, leaving 29 remaining. Five of these seven schools 
were allocated to deliver PEP to their KS3 group and two were allocated to deliver PEP to their KS4 
group, resulting in some differential attrition (leaving 17/19 schools delivering to KS4 and 12/17 schools 
delivering to KS3). Differential attrition can be (but is not inevitably) a source of bias, with a high risk of 
bias being most likely when the reason for drop-out is correlated with the outcome(s) being measured. 
The reasons for withdrawal given by the seven schools in question, however, appeared unrelated to 
attainment and included: 


e changing the school communication system from Schoolcomms and so being unable to 
commit to a full year of Schoolcomms usage (to send texts); 


e being unable to release senior staff (or commit the necessary time) to attend the training event 
at the start of the academic year; and 


e anunwillingness to divert resources away from school priorities. 


The schools that withdrew were unwilling to take part in any further aspect of the trial and so no pupil 
characteristics are available for these schools, only school characteristics. It can be seen from Table 6 
that there were noteworthy differences between the schools that remained in the study and those that 
withdrew. The withdrawn schools tended to have a poorer OFSTED rating, be smaller in size, have a 
higher proportion of pupils eligible for the Pupil Premium and to have lower attainment at GCSE. 


Table 4 describes the minimum detectable effect sizes (MDES) at the protocol, randomisation, and 
analysis stages of the study. Despite attrition—and notwithstanding the potential source of bias this 
introduces—the MDES ranged between 0.07 for absenteeism and 0.19 for science. 
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Table 4: Minimum detectable effect size at different stages of the study 


N (pupils) ICC Minimisation Power Alpha MDES 
(intervention, prognostic 
control) factors 


School 
Protocol ase sae 
(A080; 2,080) and size (34 


schools) 


School 

oes 19,298 attainment, 
Randomisation (9,631: 9,667) deprivation 
oe and size (36 


schools) 


English: 11,413 
(5,376, 6,037) 
Maths: 11,590 
(5,613, 5,977) 

Science: 10,308 
(4,726, 5,582) 
Absenteeism: 

15,355 

(7,436, 7,919) 


School 
attainment, 
deprivation 

and size (29 
schools) 


Analysis (i.e. 
available pre- 
and post-test) 


Missing data from the National Pupil Database 


Prior attainment data (KS2 data on English and maths outcomes) for all participating pupils in the trial 
was accessed from the National Pupil Database. Unfortunately, approximately 20% of this data was 
missing, particularly for those pupils in KS4 group (Year 11), which explains the higher than expected 
number of ‘not analysed’ cases reported in Figure 1. The reason for such a high proportion of missing 
KS2 data (for the KS4 pupils) is because of the school boycotts that occurred in 2009/2010 that resulted 
in many schools not submitting their test results. The NPD have confirmed this reason for missing KS2 
data and also acknowledges other instances where pupil data might be missing, for example, if a child 
did not attend school on the day of an assessment or if a child performs below the test level. 


In this sample, KS2 baseline data was more likely to be missing for pupils with the following 
characteristics: 

e KS4: 44% of KS4 pupils compared to 6% of KS3 pupils 

e SEN: 23% of those with SEN compared to 18% of those without SEN 

e EAL: 34% of those with EAL compared to 15% of those without EAL 

e FSM: 20% of those with FSM compared to 18% of those without FSM 

e Allocation: 21% of those in the intervention compared to 17% in the control group 


To formally test the extent to which missing data varied as a function of the individual characteristics 
described above and also school characteristics, a multilevel logistic regression was conducted with 
‘missingness’ as the dependent variable and individual characteristics (KS group, SEN, EAL, FSM, 
allocation) and school characteristics (school size, GCSE attainment, and proportion of children eligible 
for the pupil premium). 
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The result of this model indicated that KS group, SEN and EAL were statistically significant in their 
association with missingness. The remaining pupil-level variables (FSM and group allocation) and the 
school-level variables (school size, GCSE attainment, and proportion of children eligible for the pupil 
premium) were unrelated to missingness. Table 5 provides an explanation for the missing prior 
attainment data in the current study. 


Table 5: Description of missing reading and maths data at KS2 (baseline) 


Variable DY -X-Xeq a] o) de) a} n Percentage 
Reading KS2 Absent 32 1.03 
baseline score 
Below the test level 702 22.67 
Missing 2,355 76.04 
Malpractice a 0.03 
Unable to access test 7 0.23 
Total 3,097 100 
Maths KS2 Absent 41 1.31 
baseline score 
Below the test level 283 9.03 
Missing 2,803 89.41 
Malpractice 2 0.06 
Unable to access test 6 0.19 
Total 3,135 100 


‘Unable to access the test’ is recorded if a child is unable to sit the test because of a disability. 
‘Malpractice’ is recorded if results have been annulled due to the incorrect administration of the test 
itseli—such as children being given too long or too short a time to do it, or responses to questions being 
changed—or because of pupils cheating, and so on. Most frequently it is because of administration 
issues. 


Measurement attrition 


Post-test data was successfully collected for over 90% of the sample in English and maths. Post-test 
science data was collected for 88% of KS4 pupils but only 79% of KS3 pupils. While the number of 
cases analysed is still sufficient to maintain the power of the study, it may be the case that this 
measurement attrition has introduced some bias into the data. This was taken into account in the 
analysis of the data (See Results section for more details). 
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Figure 1: Participant flow chart 


Approached (school n=45) 


Declined to participate 
(school n=9) 


Assessed for eligibility (school n=36, 


KS group n=72, pupil n=19,298) Excluded (school n=0) 

Not meeting inclusion 
criteria (school n=0) 

Other reasons (school n=0) 


Recruitment 


Randomised 
(KS group n=72, school n=36, pupil n=19,298) 


Allocated to intervention (KS Allocated to control (KS groups 
groups n=36, school n=36, n=36, school n=36, pupil 

pupil n=9,631) n=9,667) 

Did not receive allocated Did not receive allocated 
intervention (KS groups n=7, intervention (KS groups n=7, 
pupil n=1,929) pupil n=1,421) 

Opted out (pupil n=132) Opted out (pupil n=119) 


Allocation 


Lost to follow-up Post-test data collected Post-test data collected Lost to follow-up 
(KS group n=0, (KS group n=29, (KS group n=29, (KS group n=0, 
school n=0, school n=29, school n=29, school n=0, 

=) English pupil English pupil n=6,938, English pupil n=7,365, English pupil 

n=632, maths pupil n=7,144, maths pupil n=7,388, n=762, 

e maths pupil science n=6,075, science n=6,745, maths pupil 
n=426, attendance n=7,530) attendance n=8,048) n=739, 
science n=1,495, science n=1,382, 
attendance n=40) attendance n=79) 


Not analysed (KS__}] Analysed Analysed Not analysed (KS 
group n=0, school |] (KS group n=29, (KS group n=29, group n=0, school 


n=0, English school n=29, school n=29, n=0, 

n=2,194, maths English pupil n=5,376, English pupil n=6,037, English n=2,090, 
n=1,957, science maths pupil n=5,613, maths pupil n=5,977, maths n=2,150, 
n=2,844, attend- science n=4,726, science n=5,582, science n=2,545, 
ance n=134) attendance n=7436) attendance n=7,919) attendance n=208) 


Analysis 
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Pupil characteristics 


In total, 15,697 pupils from 29 schools participated in the trial. Table 6 describes the characteristics of 
the 29 participating schools and compares them to the seven schools that withdrew post-allocation. 


The KS3 sample (n = 10,401) was twice the size of the KS4 sample (n = 5,296) and this was because 
two year groups within KS3 (Year 7, n = 5,244 and Year 9, n = 5,157) participated in the evaluation 
compared to one year group in KS4 (Year 11, n = 5,296). We would expect this 2:1 ratio to be reflected 
in both the intervention and control groups however because five of the seven schools that withdrew 
from the trial post-randomisation were allocated to deliver the programme to their KS3 cohort, this 
resulted in an imbalance in the proportion of KS3 and KS4 pupils in the intervention and control groups. 
As can be seen from Table 7, the intervention group consisted of 57% KS3 and 43% KS4 pupils and 
the control group consisted of 77% KS3 and 23% KS4 pupils. 


Table 6: Characteristics of participating and withdrawn schools 


School-level characteristics INEVa(oyare} Trial schools Withdrawn 
average? (it=y4:)) schools (n=7) 


School type 
Academy converter 1,362 (40%) 13 (45%) 3 (43%) 
Academy sponsor-led 511 (16%) 2 (7%) 2 (29%) 
Community school 657 (19%) 9 (31%) 1 (14%) 
Foundation school 299 (9%) 4 (14%) 1 (14%) 
Voluntary aided school 301 (9%) 1 (3%) 0 (0%) 


Ofsted rating 
Outstanding 681 (21%) 3 (10%) 1 (14%) 
Good 1,672 (53%) 18 (62%) 2 (29%) 
Requires improvement 656 (21%) 6 (21%) 2 (29%) 
Inadequate 173 (5%) 2 (7%) 2 (29%) 


Allocated to deliver PEP to KS3 n/a 12 (41%) 5 (71%) 


School size 
Total number of pupils ne 1,029 (337) 879 (145) 
Pupil Premium 

: a . 2 26 (1 1 
Proportion of eligible pupils 3 6 (15) 30 (16) 


School attainment proportion of 
pupils achieving A*-C GCSE English 60 59 (12) 54 (13) 
and maths (in 2013) 


English as an additional language 
Percentage of pupils with EAL = 20a”) ei) 


Special Educational Needs 
Percentage of pupils with statement 2 2 (1) 3 (2) 


of SEN or EHC plan 
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Gender 51 55 (17) 50 (2) 
Proportion of boys 


1 Based on all state-funded secondary schools in England. 


Compared to national figures (Table 6), sponsor-led academies and voluntary aided schools were 
under-represented in the trial sample, while community and foundation schools were over-represented. 
Also, fewer schools were rated as ‘outstanding’ compared to the national average. Trial schools were 
slightly larger in terms of pupil numbers, with a greater proportion of EAL pupils and a greater proportion 
of boys. Trial schools were comparable to national figures in terms of proportion of pupils eligible for 
Pupil Premium, attaining grades A* to C in English and maths, and proportion of SEN pupils or 
Education, Health and Care (EHC) plans. 


Since each school in the trial had both an intervention and a control group the school-level 
characteristics of the intervention and control groups were also balanced. At baseline, the intervention 
and control groups were relatively balanced with respect to the proportions of FSM, EAL, or SEN pupils 
within each group, however there were marginally more pupils with FSM (ES = 0.12), EAL (ES = 0.04) 
and SEN (ES = 0.05) in the control group. 


Similarly, the groups were relatively equivalent in relation to gender and baseline reading score (ES = 
0.04, in favour of control), mathematics scores (ES = 0.005 in favour of intervention) and absenteeism 
(ES = 0.05 in favour of control). 


Table 7: Baseline comparison between intervention and control groups 


Variable Intervention group Control group 


Number of KS3 pupils 4,312 (0) 6,089 (0) 


‘Number of KS4 pupils of KS4 ‘Number of KS4 pupils 3,258 (0) 258 3,258 (0) 43% 2, 2,038 (0) 2,038 (0) 28% 


ened Pupi-level = “nN (missing) “nN (missing) Percentage - nN (missing) - nN (missing) Percentage 


4, / 4,164/7570 (1) | | 4,164/7570(1) 55% 85% 4,490/8127 / 4,49018127 (0) | 55% 


a for FSM 1,048/7570 (4) 14% 1, ca 127 = 
(20) 

English as and additional | 1,293/7570 (3) 17% 1,529/8,127 19% 

language (EAL) (21) 

Special Educational 1,348/7570 (9) 18% 1,600/8,127 20% 

Needs (SEN) (19) 


White British 5,530/7570 (0) 5,427/8,127 (0) 


n (missing) Mean n (missing) Mean 
(standard (standard 
deviation) deviation) 


Reading KS2 baseline 5,944 (1626) 30.6 (9.3) 6,708 (1,419) 30.2 (9.2) 
score 


Maths KS2 baseline score | 5,972 (1598) 69.4 (20.1) 6,641 (1,486) 69.3 (20.2) 
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Absenteeism (sessions) 7,453 (117) 17.4 (20.4) 7961 (166) 16.3 (20.0) 
2013/14 baseline 


Outcomes and analysis 
Main analysis 


Table 8 reports the unadjusted post-test differences between the intervention and control groups on 
each outcome measure. The effect sizes are calculated based on the statistical model and so take into 
account any pre-test differences in English or Maths, as well as taking into account any clustering at the 
level of the school. 


In this simplest model—with only pre-test score and group allocation as covariates—the intervention 
appeared to have an extremely small, positive yet statistically significant impact on three of the four 
outcomes measured: English, maths and absenteeism. Thus, pupils who received the intervention had 
(at post-test) very slightly higher attainment in English (ES = 0.033, p = 0.024) and maths (ES = 0.067, 
p < 0.001) and slightly reduced absenteeism (ES = -0.054, p < 0.001). There was no evidence of any 
effect of the programme on science attainment. 


Table 8: Primary analysis using group allocation and pre-test scores (only) as covariates 


Unadjusted post-test means Effect Size 
Intervention Control 


Analysed 
Oiicome nalyse Mean | Analysed Mean Hedges g 


n (3>)) n (>) [95% Cl] 


0.091 0.033 p= 

english 5,376 (0.942) [0.004, 0.064] | 0.024 
sala 0.091 0.067 p< 

(0.950) [0.041, 0.092] | 0.001 
0.028 -0.013 p= 

Science 4,726 (1.009) [-0.048, 0.022] | 0.467 
0.002 -0.054 p< 

7,436 | (0.984) [-0.079, -0.029] | 0.001 


Given these very small effect sizes, the narrow confidence intervals, and the large proportion of missing 
data described earlier in the report, additional analyses were conducted to check the robustness of this 
result. To determine whether the statistical significance of the effect sizes remained under other 
conditions, additional level 1 and level 2 covariates were included in the model, specifically: 


e level 1 covariates: KS group, ethnicity, and FSM to account for group differences at pre-test 
(reported in Table 7); and 


e level 2 covariates: school size, GCSE attainment, and proportion of pupils eligible for the Pupil 
Premium. 
It was not possible to include a dummy variable representing ‘missing’ in the model because this was 
perfectly collinear with the pre-reading and pre-maths variables, however the standard errors were 
bootstrapped as an additional test of robustness. 


It can be seen from Table 9 that with the inclusion of these additional covariates the effects for maths 
and absenteeism remained statistically significant (p < 0.05), albeit extremely small. The positive effect 
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on English observed in the previous analysis has now disappeared and science remains unaffected by 
the intervention. The analysis reported below is based on bootstrapped standard errors. 


Table 9: Primary analysis using additional level 1 and level 2 covariates 


Unadjusted post-test means Effect Size 
Intervention Control 


Outcome 


Analysed Mean Analysed Mean Hedges g 
12)) (32) [95% Cl] 


0.059 -0.007 = 
(0.957) | [-0.035, 0.020] | 0.603 | ~’ 


0.007 -0.049 — 
(1.022) | [-0.077, -0.022] | 0.001 | ~” 


Multiple imputation was conducted in order to account for the high proportion of missing data. A higher 
proportion of missing data was evident in the intervention group than the control group (Chi square = 
40.08, df = 1, p < 0.001). This was because the intervention group had a higher proportion of KS4 pupils 
and it was for these pupils that the baseline data was missing, if it was missing at all. ‘Missingness’ 
(missing reading and maths KS2 data) was also negatively associated with all four outcomes at post 
test such that children whose reading and maths data were missing at baseline (KS2) had poorer English 
(ES = -0.23, p < 0.001), maths (ES = -0.17, p < 0.001), science (ES = -0.17, p < 0.001), and attendance 
(ES = 13, p < 0.001) at post-test. 


Multiple imputation 


The imputation model accounted for the clustered nature of the data (by imputing data separately for 
each cluster) and included all relevant variables and auxiliary variables involved in the analysis and 
sampling design. Data was presumed to be missing at random (MAR) rather than MCAR (missing 
completely at random) or MNAR (missing not at random). This assumption renders the missing 
mechanism ignorable, simplifying the imputation step while ensuring correct inference. Variables were 
centred prior to imputation. The imputation was performed using chained equations: this fills in missing 
values in multiple variables iteratively by using a sequence of univariate imputation methods with fully 
conditional specification of prediction equations. This method accommodates arbitrary missing-value 
patterns. Twenty imputations were conducted in order to lessen the simulation (Monte Carlo) error. 


The two complete case analyses (CCAs) reported above—the simple model (Table 8) and the additional 
covariates model (Table 9)—were then performed again, this time using the imputed dataset. Table 10 
below reports the model coefficients for the group allocation variable (4) and associated standard error 
and p value for these four models: 

1. simple model (just two covariates) using a complete case analysis (CCA); 

2. simple model using the multiply imputed dataset; 

3. additional covariates model using a complete case analysis; and 

4 


additional covariates model using the multiply imputed dataset. 
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As all variables were standardized (mean of zero, standard deviation of one) within these models, the 
beta coefficients (8) can be interpreted as an effect size. 


Table 10: Main analysis using both complete case analysis and multiple imputation 


Outcome 
1. Simple model 2. Simple model 3. Additional CR No Col iCelar-l| 
MI covariates CCA covariates Ml 


English 


Science 


Absenteeism 


Across all four analyses the results for maths and absenteeism remain largely consistent. For maths 
there is some variation in the magnitude of the effect size and the confidence intervals. Both simple 
models estimate a positive effect of the programme of 0.067. For the models utilizing additional 
covariates, the complete case analysis estimates a smaller effect of 0.029 compared to the estimation 
based on the multiply-imputed dataset of 0.039. All four effect size estimates are statistically significant. 


With regard to absenteeism, this outcome had the least amount of missing data associated with it and 
consequently the estimates of the effect of the intervention on this outcome (across all four models) is 
relatively consistent, in the region of 0.05 and statistically significant in each instance. 


For English, the direction, magnitude, and statistical significance of the effect of the intervention varies 
with each model. In the main, however (for three of the four models) it is not statistically significant. 
Similarly, for science, while the magnitude, direction, and statistical significance of the effect of the 
intervention does vary across each of the four models, a pattern of a small negative effect does emerge, 
although in most cases (again for three of the four models) this is not statistically significant. 


On treatment analysis 


The number of texts sent by schools was included as an additional covariate in the model, however 
there was no evidence that the number texts sent by schools was related to the outcomes being tested. 


Secondary analysis 
FSM analysis 


To estimate the effect of the intervention for children eligible for free school meals, the main analysis— 
including the additional pupil- and school-level covariates discussed above and using a complete case 
analysis—was repeated on the subsample of data relating to those eligible for FSM (n = 2,551). The 
results of this analysis are reported in Table 11 which shows that there was no difference between the 
intervention and control groups on any of the outcomes in this FSM subsample. As before, the standard 
errors were bootstrapped. 


Table 11: Main analysis using only the subsample of pupils eligible for FSM 
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Unadjusted post-test means Effect size 


Intervention (exe) a iige)| 


Outcome 


Analysed | Mean Analysed Mean Hedges g ae 
E : 


(10)) (3) [95% Cl] 


English : nace 068 
mBeeuteetel : (1.369) F149, 021] 


Subgroup analysis 


icc 
043 
065 
075 
052 


The potential differential impact of the intervention was explored for the following subgroups: 


e Key Stage group; 

e gender; 

e baseline score; 

e school size; and 

e EAL status (added as an additional subgroup analysis in the protocol amendment dated 15 
December 2015). 


For each outcome an interaction term was created between the group allocation variable and the 
variable representing the subgroup in question. This interaction term was then added to the main 
multilevel model (described earlier) and the associated coefficient was subsequently used in the 
calculation of the effect sizes reported below. Standard errors were bootstrapped. 


Key Stage group 


The intervention appeared to have a differential impact on pupils in KS4 compared to KS3 and this for 
both science and absenteeism at post-test. There was no evidence of a statistically significant 
interaction (between intervention and Key Stage group) for English or maths. 


KS3 pupils in the intervention groups outperformed their control group peers in science (ES = 0.07, p = 
0.01) compared to KS4 pupils where the group differences were in the opposite direction—control 
children performed slightly better than intervention children—and less pronounced (ES = -0.03). 


Figure 2: Adjusted post-test scores by Key Stage group for science 
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Conversely, for absenteeism it appears that the intervention was most effective for KS4 pupils, reducing 
their absenteeism (compared to KS4 controls) by an effect size of ES = -0.11 (p < 0.001). By 
comparison, there was very little difference between the KS3 intervention and control groups (ES = 
0.01) on this outcome. 


Figure 3: Adjusted post-test scores by Key Stage group for absenteeism 
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Gender 
There were no significant interactions between gender and group allocation on any of the four outcomes. 
Baseline scores 


Intervention pupils who had lower attainment in reading at KS2—defined as one standard deviation 
below the mean—performed somewhat better at post test compared to controls who also had low 
reading scores at baseline (ES = 0.04, p = 0.001). For those pupils with high KS2 reading, the effect 
was in the opposite direction and control pupils outperformed the intervention pupils at post-test (ES = 
-0.04). 
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Figure 4: Adjusted post-test scores by baseline attainment for English 
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There were no statistically significant interactions between baseline scores and group allocation for 
maths, science, or absenteeism at post-test. 


School Size 


For pupils in larger schools, the intervention group performed slightly better in English than the control 
group (ES = 0.02, p = 0.03). Conversely, for children in smaller schools, the control group out-performed 
the intervention group (ES = -0.04). 


Figure 5: Adjusted post-test scores by school size for English 


Intervention Control small Intervention Co ge 
small school school large school school 


English: adjusted post-test mean 


Maths attainment improved the most for intervention pupils in small schools compared to controls who 
were also in small schools (ES = 0.06, p = 0.003). For pupils in larger schools, there was only a very 
marginal difference in post-test maths scores (ES = 0.01) in favour of the control group. 


Figure 6: Adjusted post-test scores by school size for maths 


Education Endowment Foundation 30 


Texting Parents 


9 
iN) 


0.1 


Intervention Control small Intervention Control large 
small school school large school school 


i= 
Gy 
o 
£ 
~~ 
n 
o 
7 
~~ 
n 
ie} 
a 
So 
o 
~~ 
n 
s 
— 
TS 
© 
rt) 
s 
Gy 
= 


Conversely, for science, intervention pupils in /Jarger schools performed much better than controls in 
larger schools (ES = 0.19, p < 0.001). For pupils in smaller schools, the intervention pupils actually 
performed worse in science compared to controls (ES = -0.10). There were no statistically significant 
interactions between EAL and group allocation for English or absenteeism. 


Figure 7: Adjusted post-test scores by school size for science 
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English as an Additional Language (EAL) 


In the group of pupils who were not EAL, the intervention group did better in maths at post-test compared 
to the (non EAL) control group (ES = 0.04, p = 0.03). For children with EAL status the opposite effect 
was observed such that the control group performed better than the intervention group in maths at post- 
test (ES = -0.04). There were no statistically significant interactions between EAL and group allocation 
for any of the other outcome variables. 


Figure 8: Adjusted post-test scores by EAL status for maths 
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Parent outcomes 


The project delivery team collected data on parent outcomes via a phone survey from 2,891 intervention 
and control families (18% of the overall sample). Approximately 10—20% of participating families were 
sampled from each of the 29 schools in the trial. 


The majority of respondents (92%) identified themselves as the child’s parent and 80% identified their 
child’s ethnicity as White British. Twenty-six per cent (26%) of the sample were educated to degree level 
or above and the majority of respondents (92%) identified English as the primary language spoken at 
home. 


The pupil characteristics of the surveyed sample are compared to those of the main sample in Table 
12. It can be seen that the survey sampled was largely representative of the main sample. 


Table 12: Comparison of pupil characteristics between main sample and surveyed subsample 


WETar-1e) (=) Main sample Parent survey subsample 
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Mean Mean 
(standard (standard 
deviation) deviation) 


Reading KS2 baseline 12,652 30.4 (9.2) 2,400 30.9 (9.2) 
score 


“Maths KS2 baseline score. KS2 baseline score | 12613 | 613  69.3(20.1) | 3  69.3(20.1) | 1) 2386 | 386 70.8 (20.0) | 8 70.8 (20.0) | 0) 


oe (sessions) 15,414 16.8 (20.2) 2,811 15.5 (17.7) 
2013_14 baseline 


Parent engagement 


Parents were asked ten questions about their engagement in their child’s learning which were combined 
into a single measure called ‘parent engagement’ (See section on Outcome Measures for more detail). 
In addition, parents were also asked four questions (to which they could respond ‘yes’ or ‘no’) about 
what kind of school-related information they discussed with their child: 


e Have you asked your child what they learned at school? 

e Have you talked with your child about their attendance? 

e Have you asked your child to revise for an upcoming test? 
e Have you asked your child about their grades? 


Table 13 below reports the models for each of these five parent outcomes. The same covariates that 
were used for the main analysis were also used in these analyses. 


There was no evidence that the programme improved outcomes for intervention families in relation to 
parent engagement or that it increased the frequency with which parents talked to their children about 
what they learned at school, their attendance, or their grades. It is important to note, however, that the 
vast majority of parents, both intervention and control, were scoring very highly on these outcomes (with 
the exception perhaps of talking about attendance) and it may be the case that there was little room for 
improvement to be made. 


However, the odds of parents in the intervention group talking to their child about revising for an 
upcoming test were almost three times greater than the odds of control parents talking to their child 
about revising—an odds ratio (OR) of 2.811 (95% Cl [2.070, 3.817], p < 0.001). This statistically 
significant result indicates that intervention parents were more likely than controls to talk to their child 
about revising for an upcoming test. (An OR of 1 indicates no effect of the intervention, an OR greater 
than 1 indicates that the effect is in favour of the intervention group, and an OR less than one indicates 
the effect is in favour of the control group.) 
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Table 13: Analysis of parent outcome data 


Unadjusted means Effect Size 
Intervention Control 


Outcome (SD) [95% Cl] 


Parent 1.362 -.052 : -.040 
engagement : (.947) : [-.116, .035] 


Intervention (exe) aiige)| 


Analysed | Mean Analysed Mean Hedges g 
n 


Analysed 
n 


Analysed Odds ratio 


% % 
0 yes © yes [95% Cl] 


Outcome 


a} 
Has parent 
asked child 
what they 1,355 1,501 ee 178) 
learned at : — 
school? 
Has parent 
talked with 6 .956 
child about | 9°! 1808 67% | £807, 1.133] 
attendance? 
Has parent 
asked child to 2811 
revise for an 1,356 1,494 88% : 
upcoming [2.070, 3.817] 
test? 
Has parent 
asked child ' 1.024 
about their £200 4201 ad [.773, 1.355] 
grades 


1 Mixed effects logistic regression was used to analyse dichotomous outcomes. 


Cost 


The cost of the programme as delivered in this one-year trial is estimated at £7.55 per pupil. This figure 
includes the cost of teacher training as well as costs associated with implementing the programme (both 
project delivery team support and text message costs). 


Teacher training 


Schools required one and a half days of supply cover to enable a representative to attend the training 
sessions (one full day in September and one half day in January). Not all schools will have required 
cover, however and at an estimated cost of £200 per day this will have cost each school approximately 
£300 per teacher. 


Project delivery team support 
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The project delivery team employed a number of research assistants (RAs) to support the 
implementation of the programme within schools. Each RA cost approximately £20,000 for the duration 
of implementation and supported eight schools. This equates to a total cost of approximately £72,500 
across 29 schools—£2,500 per school. One school did allocate paid teacher time to support the 
intervention and this cost the school in the region of £2,000. 


Text messages 


Once teachers are trained to deliver the programme the ongoing costs to the schools are the text 
messages in addition to an annual Schoolcomms license fee of £1,500 per annum. Parents should be 
sent an average of 65 texts per year, and at £0.05 per text this is the equivalent of £3.25 per child. 


Thus, for a typical secondary school with 1,000 pupils, the cost to the school is: 


e £3.25 x 1,000 (for 65 texts to 1,000 pupils); 

e £2,500 for project support; 

e £300 to cover one teacher to attend 1.5 days training; and 
e £1,500 annual Schoolcomms license fee. 


This comes to a total of £7,550 per school (£7.55 per pupil) for the first year of implementation, with 
ongoing yearly costs relating to the cost of the text messages (£3.25 per pupil) and the annual 
Schoolcomms license fee (£1.50 per pupil). Table 13 summarises cumulative and average costs per 
year. 


Table 13: Cost per year over three years 


Number of years using the Cumulative cost per pupil Average cost per pupil per 
programme Werle 
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Process evaluation 


This section presents the key findings from the process evaluation. It involves the findings from an online 
survey with school personnel (n = 123), interviews with headteachers and teachers (n = 12), four focus 
groups with pupils (n = 29), and a telephone survey of 1,365 intervention parents. 


Interview and focus-group discussion recordings were transcribed verbatim. Data was thematically 
content analysed. Transcribed data was read and re-read enabling the analyst to become immersed in 
the experiences and views of the informants, and to enable themes to be extracted inductively. Data 
was organised by emerging and recurring themes: those which arose spontaneously and were 
consistent between informants (unless otherwise stated) have been presented. 


Implementation 
Training and Support 


School personnel were asked about training and support as part of the survey. Overall—following the 
initial training event in September 2014—the majority of respondents (71.6%) were positive about the 
training received, and thus confident (70.7%) about implementing the programme. 


On reflection in May—June 2015, this was echoed in the interviews where school personnel expressed 
satisfaction with the continuous support received throughout the delivery of the programme. The 
comments below are representative of the teachers’ views after the initial training. 


‘| think they were really supportive. If we were ever, we’d phone [Research Assistant from the 
project delivery team] up and he’d talk us through it on the phone. He was really good so | don’t 
think there was anything extra that we really needed.’ 


‘The [Research Assistants] were really good at always being available on the phone and they 
came ina couple of times. Yeah, so | felt like there was a lot of support.’ 


Only one of the schools reported a problem with the continuous support received during the programme 
implementation: 


‘But it became a little bit, kind of, hit and miss as to when we’d get them (emails). Whereas they 
started off quite regularly, they started becoming not very regular and then not at all, so it was 
very difficult to know when our week [for conversation prompts] was. So ! think the 
communication was probably the thing, rather than just the training. The thing is, he [project 
delivery team RA] left his post, kind of, mid-year. | think it was just before Easter or somewhere 
around there, and it kind of fell off a little bit then, really. | don’t think we’ve really done very 
much since then.’ 


Barriers to delivery 


A number of barriers to delivering the PEP programme are reported, and these fell broadly into two 
categories: (1) acceptance of the programme, and (2) organisational constraints. 


The initial challenge for schools in implementing the programme was gaining teacher willingness to 
participate. Despite 72.3% of those who completed the survey acknowledging that implementing the 
programme has been beneficial for their school, each school interviewed mentioned problems with 
getting teachers on board at the beginning. 


‘| think one of the problems was that there’s quite a lot of people involved in the projects because 
it’s all the maths and English and science departments that were involved and it was kind of 
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getting them on-board which was the tricky part of the project. So we weren’t wholly prepared 
for that.’ 


All teachers interviewed referred to various organisational constraints being a challenge to successful 
implementation. For example, the main barrier to delivery of the programme, mentioned in all the 
interviews, was reminding to upload content data for the various text messages. 


‘Well, as a classroom teacher, for me, not a great deal of challenge because all | had to do was 
write a text message. The greater challenge was making sure | did it, you know, and reminding 
myself that I’ve got to do this text message, or I’ve got to input the date.’ 


There was also criticism surrounding text messaging causing confusion to parents. This was due to two 
reasons: schools uploading incorrect information, and the restrictiveness of text message character 
length. A typical comment regarding the former was: 


‘The type of information that can be sent, because it needs to be very, very short... because of 
that, sometimes the understanding of the parent as to what the message means and the 
importance of stuff gets a bit confusing.’ 


‘The only issue begin was our end is when staff entered incorrect information or was conflicted 
information. Because that then generates a text straight out, that sometimes caused a little bit 
of a problem. One example being because of staff weren’t completely aware how the system 
went on its initial stages, we had a group of parents were contacted to say that they went ona 
behavioural report, which obviously was never going to be the case, but because it was the first 
dropdown on the defaults, that’s what they were sent.’ 


It was acknowledged that because the programme was in the development phase it was slow to 
commence: 


‘If it operated as it should have operated, | had anticipated they [parents] would be getting one 
[text] a week. | think there were some problems to overcome in the first half-term so | don’t think 
they got it off as fast as they wanted. So | think there was probably the first four weeks where 
there wasn't probably very much.’ 


Furthermore, pressures of school life, at times, impacted on the programme. One teacher said: 


‘...English, maths and science are core subjects, you’re always being asked to do extras, this 
was just another one on the pile, so it’s not a lot to do, and | know it sounds like a lot, but fitting 
it in somewhere and finding something like a good conversational topic was tricky actually and 
then trying to make sure that the prompt and statement all lined up on the dates for all the 
classes and things like that was, in the evenings as well.’ 


Another problem was that school’s found it difficult working with a third party (the project delivery team 
RA), which meant that schools did not have sole control. 


‘ think ... it was just working [with] a third party, | found that quite challenging. So people were 
kind of saying to me, ‘Don’t do this’, and then, you know, having to deal with another person all 
the time and... And sometimes there were email conversations going on between [project 
delivery team RA] and [school liaison officer] and one of the curriculum leaders that | wouldn't 
know about. And it wasn't, you know, anything bad, it was just that then you don’t have that full 
control of what’s going on, it was hard to have an overview. And obviously | think they had a full 
overview but it would have been nice if I'd had a full overview.’ 


A number of school-level organisational issues emerged surrounding the implementation of the 
programme. At the beginning, schools typically experienced problems securing the commitment of 
teachers to the programme and its schedule leading to problems with the timely uploading of text 
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content. Clearly, the lack of control that school’s experienced when participating in the programme may 
have had a significant impact on their engagement. 


Fidelity 


In every school the project delivery team monitored the number and type of texts sent to parents for 
each subject—English, maths and science. As previously described, the type of text message included 
advance notice of upcoming assessments, conversational prompts, and notice of missing homework. It 
was intended that parents would be sent around 65 texts in total over the course of the trial; on average 
however, the total was 30. There was, however, considerable variability within this, from a minimum of 
15 to a maximum of 77. The average number of messages sent per subject was relatively consistent at 
around 16 or 17: for English the average was 16.3, for maths, 17.5, and for science, 16.0. 


It transpired that texts reporting absenteeism did not take place—these were not sent as part of the 
intervention by either the project team or by schools. From its previous feasibility study, the project 
delivery team found that parents were irritated if they received texts from the school relating to their 
child’s legitimate absence(s). For this reason the intention was that parents would only be alerted to 
illegitimate absences. The difficulty was, however, that attendance data was cleaned—legitimated 
absences were updated on records—after a delay of two weeks, making the data more complicated to 
extract in a timely manner, and in some cases cleaning issues occurred such that the wrong parents 
were texted. Furthermore, since the rule revision in September 2013 prohibiting schools from authorising 
absences, attendance figures have significantly reduced. With this in mind, the project delivery team 
decided not to overburden schools by requesting this data and instead focused on collating test-date 
information. 


It would appear that schools sent text messages regarding upcoming assessments (M = 22.19) much 
more frequently compared to missing homework (M = 11.39) and conversational prompts (M = 5.08). 
This was also the case for KS group: a higher number of texts detailing upcoming assessments was 
sent within the KS4 cohort (M = 12.35) than the KS3 cohort (M = 9.32). This was reflected in the process 
evaluation interviews during which school personnel described some of the difficulties they encountered 
with conversational prompts, for example, drafting information that was both clear and suitable for 
parents while also adhering to the constraints of the text character limit. This important point is reflected 
in the comments below: 


‘It was a bit short actually, it was really hard to get some of that down to the correct size, and 
parents more often than not were confused. ’ 


‘1 don’t think the conversational prompts are easy to implement in any subject. | think 
assessment dates, as I’ve said, probably easier to get them out for some subjects than for 
others, and it will be much closer to the assessment that we can send that information out for 
some subjects like English.’ 
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Table 14: Mean number of text type sent per subject 


Subject 


Type of text Mean (SD) 
Upcoming assessments 6.96 (2.42) 
Missing homework 3.79 (3.58) 
Conversation prompt 2.08 (1.56) 
Upcoming assessments 7.64 (2.87) 
Missing homework 4.38 (4.88) 
Conversation prompt 1.64 (1.52) 
Upcoming assessments 7.59 (3.91) 


Missing homework 3.22 (4.42) 


Conversation prompt 1.36 (1.40) 


Schools also expressed the view that a few of the texts, particularly those concerning class tests— 
where pupils were not required to revise—were not suitable and should not have been sent to parents. 


‘But sometimes with the tests, some of the tests didn’t really need a text message. So like, for 
example, we did the assessment at the end for the project so we had to do the assessment at 
the end and we had to send a text message out about that assessment but the children couldn't 
do anything for the assessment, they couldn't revise for it.’ 


There was some criticism concerning the frequency of texts sent to parents, especially to those having 
more than one pupil in receipt of the intervention, which led to some parents making complaints to the 
school. 


‘Yeah, well that was the thing as well, is that sometimes they received like four in a week and 
then sometimes they didn’t receive any and | received a couple of complaints about that, and 
particularly if you have a child in Year 7 and Year 9, and particularly we've got some children... 
you know, parents that have got twins, so, you know, if you’ve got twins in Year 7 and a child in 
Year 9 you're... their phone, they said it was almost like beeping constantly because if you just 
send four text messages, well then that becomes twelve text messages, and then other times 
there'd be no text messages, So...’ 


Despite this, in the school and teacher survey, the majority of respondents (75.6%) agreed that the 
texting frequency was sufficient to impact on parents’ behaviour and attitudes. 


Outcomes 


While 70% of school personnel believe that the programme engages and motivates the parents, there 
was a range of opinions on whether the programme would benefit students in terms of their parental 
engagement. A minority noted they found it difficult to see the outcomes for students, as ‘it’s difficult to 
know what the conversations parents have been having with the students at home.’ 


On the whole, schools did feel that the programme has resulted in increased communication between 
school and parents: ‘we have communication with our parents much more than we normally do’, and 
one school did comment that attendance numbers at parents evening have risen: 


‘If we look at attendance and you know, parents’ evening and the like, and other events that 
would be you know, statistics show that the attendance has gone up in Year 11 parents. And 
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there’s Statistics for everything that we do so we can compare year on year so that’s quite 
encouraging, that we can identify as some of the parents who we have previously found very 
difficult to make contact with or get involved in any way with the school, some have, so | don’t 
know, | have to suggest that might be down to the programme.’ 


Positive consequences of the programme, which were widely mentioned, include the programme’s 
ability to develop and promote a three-way relationship between school/teacher, parents, and pupils. 
This was achieved by breaking down barriers between school/teacher and parents while stimulating 
communication between parents and pupils in the home environment, reflected in the comments below: 


‘On the whole, I’d say that it has had a positive influence because, certainly, my Year 9 students 
have come back and said, oh yeah, | had a conversation about Romeo when he tells Juliet he 
loves her for the first time. You know, | talked to my mum and dad about that. And | think that’s 
really pleasing to know that what you're doing is reaching beyond the classroom and their 
parents are being made aware of it, but actually, not just aware of it, being involved in it.’ 


‘It’s empowering parents to talk about learning, not just talk about behaviour, which sometimes 
takes up too much of a focus.’ 


‘| think my staff would say that they very definitely appreciate having parents on their side as 
well, having a way of communicating with parents, adult-to-adult, so that there is a team around 
the child, a team around the student, trying to get the best out of them.’ 


Other positive consequences, for the school, included text messaging being cost effective and time 
effective: 


‘Well, | just think that we are much more heavily text orientated than we use to be. We’re sending 
many, many less letters out than we used to and so our postage cost has gone down 
significantly. But staff are talking to parents much more regularly by text.’ 


‘It’s made life a little easier around the fact that low level stuff you're not on the phone constantly 
all night trying to get hold of parents. | can usually say, yes I’ve contacted that parent by just 
clicking a button. And that is, when you’ve got a hundred other things to do, its really quick, it’s 
really easy. It allows us to tick a box to say, yeah, we’ve done that bit. And I’ve really appreciated 
it, definitely.’ 


Pupils’ experience of the programme 


All pupils participating in the focus groups were aware of the programme and the type of information 
parents received via text messages from their school. Opinions were consistent regarding their 
experiences of PEP. While pupils acknowledged the programme was, at times, ‘helpful’ and ‘effective’, 
it was often referred to as ‘annoying’. There were three main reasons for pupils branding the programme 
as annoying. First, pupils felt that it limited their independence as parents now were aware of upcoming 
assessments and so on, resulting in parents ‘reminding’ and ‘nagging’ them to prepare. Second, it 
caused ‘frustration’ at home (on the pupil’s part) when incorrect information was sent: parents would 
remind children of the content received in a text which at times wasn’t correct. Lastly, lack of clarity in 
the text messages caused confusion at home: pupils stated that they weren’t detailed enough—for 
example, a text might have informed parents of an upcoming assessment without giving any information 
about the type of assessment it was (a class test or an exam). 


Aside from this, pupils were in agreement that the programme did benefit their parents’ relationship with 
the school: 
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‘They have more of a connection with the school, like because obviously they don’t come to the 
school with us so it’s ways of finding out what we’re doing because otherwise like if you don’t 
really talk to your parents they wouldn't find out.’ 


It also enabled communication between parent and child as it gave depth to their conversations 
surrounding school: 


‘| think it just means that they partially understand what I’ve been doing so | don’t have to explain 
it all from like...it makes it easier to talk about what you’ve been doing at school.’ 


Acceptability of the programme to parents 


Overall, parents interviewed (n = 1,365) were happy with the programme in relation to the content, 
frequency, and timing of texts. They found the texts easy to understand (> 90%) and reported that they 
typically shared the information received with the child’s other parent or carer (62%), with another 
relative (9%), or with a family friend (8%). Interestingly, just over a quarter of parents interviewed (27%) 
said that they did not share the information with anyone. 


Types of texts received 


Parents were asked how frequently they talked to their child about the different types of messages they 
received. Ninety-six per cent of parents reported receiving texts about upcoming tests, and the majority 
indicated that in response to this type of text they talked to their child about their test—either always 
(69%) or most of the time (18%). A smaller proportion of parents (70%) reported that they received texts 
about what their child was learning in school and that they talked to their child about it —either always 
(46%) or most of the time (14%). 


The majority of parents reported that they did not receive texts about their child’s grades (73%), however 
18% said that they did and consequently they always talked to their child about their grade in that test. 
Similarly, 74% of parents interviewed reported that they did not receive any texts about missing 
homework (which may be because this type of text wasn’t sent or the child rarely had missing 
homework). Nineteen per cent of the sample said they did receive texts about missing homework and 
as a result they always talked to their child about this. 


These results reflect the variability in texts reported in the fidelity section above, with texts about 
upcoming tests and what was taught in school that day being the most frequent type of message 
received by parents. Consistently, however, of those parents who did receive alerts (regardless of the 
content), the majority responded by talking to their child about it either always or most of the time. Only 
a very small minority of parents (less than 2%) rarely or never talked to their child about the texts they 
received. 


Child’s response 


When parents were asked how their child responded when they brought up information contained in the 
text messages, about half (48%) indicated that their child’s response was neutral, 39% said that their 
child’s response was somewhat or very positive, and a minority of parents (12%) reported that their 
child’s response was either somewhat or very negative. 


Usefulness, frequency and timing of texts 


When asked which two types of texts they found most useful, the overwhelming majority of parents 
reported that they found the texts about upcoming tests the most useful (85%) followed by texts about 
what was taught in school that day (42%). 


Most parents felt that the number of texts they received during the course of the year was about right 
(84%) with a minority reporting that there were too many (10%) or too few (6%). Equally, parents 
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preferred to receive the information by text (97%) rather than email (45%), smartphone app (22%), 
Facebook message (14%), a note sent home with the student (9%), or by post (7%). 


During the trial texts were sent at approximately 4pm and parents were asked whether this or another 
time of day would be more convenient, however over 94% of parents were happy with the timing of texts 
at 4pm. With regard to upcoming tests, parents were informed four days in advance of the test and also 
the day before the test. Fifty-one per cent (51%) of parents responded to say that they would prefer to 
get a single text four days in advance, however 41% said that actually they preferred to get both. The 
majority of parents indicated that four days’ notice (68%) was sufficient although a significant minority 
would have preferred seven days’ notice (20%). 


Formative findings 


Overall, in this first year of implementation, schools enthusiastically embraced the PEP programme. 
There were positive comments about the use of text messaging for engaging parents, praised for its 
‘immediate, non-intrusive way of getting in touch with a significant number of parents’ and ‘informal way 
of contacting parents’, while being cost-effective to schools: ‘We now find texting much cheaper than a 
stamp.’ Our survey revealed that 93.5% of school personnel agreed that text messaging is a suitable 
way to communicate information to parents. Similarly, parents who were interviewed reported being 
happy with the content, frequency, and timing of texts and indicated that they subsequently discussed 
the information contained within the texts with their child. 


On the whole, schools report a positive implementation experience with 75.6% of those surveyed 
recommending the programme to other schools. While schools would be willing to support the roll out 
of the programme in subsequent years (72.4%), the following recommendations from the perspective of 
schools have been suggested. 


Training 


One school suggested that the initial training sessions be made available online (through a webinar) so 
that all of those involved in implementing the programme could benefit from the training sessions at a 
time convenient to them and at no cost (Such as travel costs). While the project delivery team did make 
video recordings of the training available to all schools, it seems that not all schools were aware of this. 


Targeted delivery 


While schools acknowledge that it may be difficult to implement such a programme for every pupil in 
attendance due to teacher burden (additional workload) and parent burden (the receipt of too many 
texts), three of the schools did suggest that PEP would be most useful for certain groups, for example 
Pupil Premium students or hard to reach parents. 


‘If you were to target Pupil Premium students and hard to reach parents, | think it would be very 
successful and important part of the whole range of things that we use to sort of support them 
through...’ 


Organisation and Management 
A number of suggestions for improvements to the programme have been identified by schools: 


1. Schools feel parents should be better informed at implementation, and made aware of the 
texting strategy, specifically the content that they can expect to receive during the academic 
year. This would help to overcome any confusion that was experienced by parents. 


2. Schools also suggest that, at their end, they have an administrative procedure in place to 
monitor all text content, to overcome any incorrect information being sent to parents and to 
oversee the scheduling of the text messages. This would help to ensure that parents receive 
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the information at the correct time, for example, assessment prompts could be sent a week 
ahead and not three days before. 


3. Schools also identify the need for a ‘lead’ within each subject that the programme is running 
for. This would help to ensure reminders are sent, and that consistency of the programme is 
maintained. 


4. Lastly, schools feel that the assessment prompts would be better received if they were 
followed up with grades. 


Recommendations surrounding organisation and management of the programme are reflected in the 
comments below: 


‘We could almost do with sending a letter out at the beginning of the year to say, look, you will 
be notified at certain times over this type of thing and this type of thing. And just perhaps making 
the message a tiny, little bit clearer; there will be an assessment on this. Your student need to 
revise this, or doesn’t need to do this, or just that tiny, little bit more. And I think that would be, 
you know.’ 


‘If there was a teacher in each department who was, like, the lead for it, | think that would be 
quite helpful. Because it is very, very quick and easy, it’s just reminding ourselves.’ 


‘| think probably what would have been better, you know, if it were carrying on, to pick up some 
of the grades because then to be able to, after the tests, text home and say, well, this is the 
grade that your child got in the test.’ 


‘The scheduling of texts to make sure that they happen at the best possible time for parents to 
be able to effect a change and maybe even having a couple of texts building up to things like 
tests so you tell them the date of it early and then you remind them.’ 


Control group activity 


As the trial was designed with randomisation at Key Stage level, the risk of contamination between 
groups was low, and there was no evidence that any pupils in the control groups were exposed to PEP. 
Furthermore, the process evaluation did not identify any spill-over effect to the control group pupils. 


Schools (n = 4) were questioned (through interviews, n = 12) about communication with parents in their 
control groups. This indicated that those parents were not aware of the intervention, and communication 
from the school was ‘standard practice-——Schoolcomms, email, telephone calls, letters, and pupil 
homework diaries/planners. 


‘It was standard practice, email mainly, but they would have received way less information.’ 


‘So for things like test timings and things, they would have had that as part of our original 
calendar and through their student planners and the like, but wouldn’t have had a notification 
necessarily that it was coming in the way that the treatment group would have done.’ 
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Conclusion 


Key conclusions 


1. Children who had the intervention experienced about one month of additional progress in maths 
compared to other children. This positive result is unlikely to have occurred by chance. 


2. Children who had the intervention had reduced absenteeism compared to other children. This 
positive result is unlikely to have occurred by chance. 


3. Children who had the intervention appeared to experience about one month of additional progress 
in English compared to other children. However, analysis suggests that this finding might have 
been affected by bias introduced by missing data, so we cannot reliably draw this conclusion. 
There is no evidence to suggest that the intervention had an impact on science attainment. 


4. Schools embraced the programme and liked its immediacy and low cost. Many respondents felt 
that the presence of a dedicated coordinator would be valuable to monitor the accuracy and 
frequency of texts. Schools should consider whether they would be able to provide this additional 
resource. 


5. The vast majority of parents were accepting of the programme, including the content, frequency, 
and timing of texts. 


Limitations 


There were a number of limitations to the current study. The first of these was the assessment of science 
attainment at KS3. Unfortunately it was not possible to use an existing reliable and valid measure of this 
outcome as none were available and the evaluation team did not have sufficient time or resources to 
develop and pilot their own assessment measure prior to post-testing. As such, it was necessary to rely 
on an old (2009) SAT science test that perhaps had limited content validity and/or constructive 
alignment. This could have led to the imprecise measurement of science attainment, and indeed, while 
a small positive effect was observed for English and maths, no such effect was observed for science 
and poor measurement might be the reason. Despite this, however—and the potential to observe a 
basement or floor effect—there was considerable variability in the KS3 science scores, and these were 
approximately normally distributed suggesting that the measure was, in fact, capable of detecting 
variability in scores and any potential group differences. It should also be remembered that this was the 
measurement tool for two thirds of the sample and that GCSE scores were also used to measure science 
attainment (for KS4 pupils). For these reasons we do not believe that the measurement of science 
attainment at post-test was overly compromised. 


A second potential limitation was the withdrawal of seven schools post allocation. This differential 
attrition resulted in an imbalance with respect to the proportion of KS3 and KS4 groups (and therefore 
also missingness) in the intervention and control groups which may have introduced a source of bias 
between the groups and acted as a confounding variable on the outcomes being measured. For this 
reason a number of additional covariates were included in the main (multilevel) regression models to 
control for group differences at baseline in terms of FSM, ethnicity, and Key Stage group as well as 
school-level differences including school size, attainment, and proportion of pupils eligible for the Pupil 
Premium. 


Finally, an unexpected issue arose around the proportion of KS2 (baseline) English and maths data 
missing from the NPD dataset for the KS4 cohort. The reason for the missing data is the school boycotts 
that took place in 2009/2010, however the potential implication of this for the reliability and credibility of 
the results (related to English, maths and science) meant that it was necessary to use multiple 
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imputation to try and account for this missing data and obtain a more reliable estimate of the effect of 
the PEP programme on the measured outcomes. 


Notwithstanding the limitations described above, this was a large trial that had sufficient power to detect 
small effects. It does need to be acknowledged, however, that due to the proportion of missing KS2 
data, the differential attrition of seven schools at the start of the study, and the non-random sampling of 
schools, the external validity of this study is somewhat compromised and we should be cautious in terms 
of generalizing these results more widely. 


Interpretation 


This trial aimed to measure the impact of the PEP programme on English, maths and science attainment 
and absenteeism. Using a simple two covariate (multilevel regression) model and a complete case 
analysis, we found extremely small yet positive effects of the intervention on three of the four outcomes, 
namely English (ES = 0.034, p = 0.026), maths (ES = 0.067, p < 0.001) and absenteeism (ES = -0.054, 
p = 0.001). It should be noted that such small effects were only detectable because of the large sample 
size. When the same (simple) analysis was conducted using multiple imputation, the effects for maths 
and absenteeism remained consistent in magnitude and statistical significance, however the effect on 
English disappeared and a small negative effect on science emerged (ES = -0.034, p = 0.022). 


When additional covariates were added to the model (in a complete case analysis) the effect of the 
programme on Maths (ES = 0.029, p = 0.033) and absenteeism (ES = -0.049, p < 0.001) remained 
intact, albeit the effect size for maths reduced in magnitude from 0.067 to 0.029. The effect of the 
programme on English, however, disappeared as it did with the MI analysis of the simple model reported 
above. When this additional covariate analysis was conducted again, this time using multiple imputation, 
the results were consistent with the complete case analysis. That is, there was a (slightly larger) effect 
of the programme on maths (ES = 0.039, p = 0.003) and the same magnitude of effect as before on 
absenteeism (ES = -0.050, p < 0.001). There was no effect of the programme on either English of 
science in this scenario. 


The results of this evaluation are commensurate with other studies which have found that providing 
parents with regular, concise updates regarding their child’s progress, missing homework, and exam 
performance can improve academic attainment (Bergman, 2014; Kraft and Rogers, 2015; York and 
Loeb, 2014). However, the current trial provides evidence of only a very small positive effect and was 
unable to confirm the larger effect sizes—in the region of one fifth to one third of a standard deviation— 
reported by these other studies which tended to have smaller sample sizes (< 500), were more intensely 
delivered, and not independently evaluated. Another possible issue to consider is that the smaller effect 
sizes detected in this trial might be the result of insufficient differentiation between the intervention and 
control groups. Given that the Schoolcomms system is used routinely by schools anyway (albeit not in 
exactly the same way for the control children), there is a risk that the difference between the control and 
intervention activities was insufficient to elicit larger changes in the measured outcomes. 


Inthe process evaluation, teachers suggested that the intervention would be more feasible to implement 
if it were targeted (rather than universal) and there is certainly some support for this from the trial data. 
While the intervention worked for the average child in terms of consistent but small improvements in 
maths and absenteeism, PEP seemed to work better for certain groups of pupils across the different 
outcomes. (It should be noted that any group differences reported in the FSM subgroup analyses were 
not statistically significant.) 


While there was some, inconsistent evidence that PEP improved attainment in English for the average 
child, there did seem to be a positive effect at post-test for pupils who had lower attainment in reading 
at baseline (ES = 0.04 for poor readers at baseline compared to ES = -0.05 for good readers at baseline). 
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In maths, the children who made the most progress (compared to the control) were those who received 
the programme and did not have EAL (ES = 0.04). For children with EAL status, the effect was in the 
opposite direction and those in the control group performed better at post-test than those in the 
intervention (ES = -0.04). Interestingly, maths attainment was also better for those intervention children 
who attended small schools (ES = 0.06) compared to intervention (and control) pupils in larger schools 
(ES = -0.01). 


The results for science were mixed: for the sample as a whole there was no consistent evidence that 
the programme improved science attainment for the average child. However, KS3 pupils seemed to 
benefit more from the programme (ES = 0.06 vs controls) compared to KS4 pupils who also received 
the programme (ES = 0.01 vs controls). Improved attainment in science was also observed for 
intervention children in larger schools (ES = 0.19, compared to control pupils in larger schools); 
intervention children in smaller schools performed worse in science than comparable controls (ES = - 
0.10). 


There were some effects of the programme in relation to a reduction in absenteeism. Given that no 
absence-related texts were in fact sent, this is a particularly curious finding which is perhaps related to 
the increased monitoring of children’s school-related activities overall, creating an environment in which 
pupils felt less able or willing to truant. In particular, the intervention was more effective in reducing 
absenteeism in the KS4 cohort (ES = -0.11) compared to KS3 (ES = -0.01), and this result is of particular 
note given that absenteeism increases as pupils progress through school, is more likely if parents do 
not value education, and is closely linked to lower academic attainment (Taylor, 2012). 


Of the parent outcomes that were measured, the only detectable (positive) effect was in relation to the 
likelihood of parents asking their child about revising for upcoming tests, where intervention parents 
were more likely to do so than control parents (OR = 2.81, 95% Cl [2.070, 3.817)). 


While the effects of the intervention were very small, the programme is inexpensive and relatively easy 
for schools to implement. Parents reported general satisfaction with the frequency, content, and timing 
of the texts and in the majority of cases talked to their children about the information they were receiving 
via text from the school. Similarly, the teacher survey and interviews showed that schools were 
enthusiastic about PEP and liked its immediacy and low cost. While teachers acknowledged that there 
would need to be a person (or persons) responsible for its coordination within the school—to monitor 
text content, accuracy, and frequency—they could see the value of the programme, especially if it was 
targeted at specific groups of pupils and parents perceived as ‘hard to reach’. 


There was considerable variability between schools in terms of the number and type of texts sent— 
some schools struggled with the more complex and time-consuming texts (conversational prompts) and 
these were sent least frequently by all schools. Even so, this trial detected small and consistent 
improvements in maths and absenteeism, and while it is not possible to determine what type or 
combination of texts might be most effective, the level at which PEP was implemented in the current 
study does suggest that this programme can make a small, positive difference to some academic 
outcomes. 


Given the government’s increasing focus on supporting parental engagement, this programme might 
well have a promising future. The effects detected by this trial are so small, however, that in combination 
with some of the external validity issues raised elsewhere (relating to missing data, sampling, and 
differential attrition) it is not yet possible to conclude with any certainty that this programme is effective 
in improving academic outcomes for pupils. 
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Appendix 1: Phone interview questions 


Q00. Hello. My name is [CALLER NAME]. University of Bristol and [SCHOOL NAME] are working 
together to learn how parents can be helped to support their child’s learning. May I please speak 
with [PARENT NAME] and could you please confirm you are the parent or carer of [STUDENT 
FULL NAME]? 


00 No, is not [PARENT NAME]. 


Thank you for your time. Have a good [day/evening/night]. Goodbye. [Terminate 
survey] 


01 Yes, is the parent carer > Go to Q1 
90 Don’t Know 
91 Refused 


92 Terminated 


Q1. Okay, thank you. | am calling you today to ask how you support [CHILD NAME’S] schooling. 
The survey will take approximately 5 minutes. The school won’t be given access to your 
individual answers and you won’t be identifiable in any research published by University of 
Bristol. You do not have to participate, you can choose to not answer any questions | ask and 
you may stop at any time. Are you happy to take the survey? 


00 No 
Thank you for your time. Have a good [day/evening/nigh]. Goodbye [Terminate 
Survey] 


01 Yes > Go to Q2 
91 Refused 


92 Terminated 


[parent_beliefs] 


Now | will read a series of statements. Please respond to the question based on how strongly 
you agree with each of the following statements. Please use the following options: strongly 
disagree, somewhat disagree, somewhat agree, strongly agree. [ORDER of Q2-Q5 (inclusive) IS 
RANDOMIZED] 


[Role_construction1] 
Q2. It is my responsibility to make sure [STUDENT FIRST NAME] attends school every day. 


1 Strongly Disagree 
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Somewhat Disagree 
Somewhat Agree 
Strongly Agree 
Don’t Know 


Refused 
Terminated at this question 


[Role_construction2] 


Q3. It is my responsibility to make sure [STUDENT FIRST NAME] submits all of [HIS/HER] 


homework. 


1 


2 


3 


Strongly Disagree 
Somewhat Disagree 
Somewhat Agree 
Strongly Agree 
Don’t Know 


Refused 
Terminated at this question 


[Role_construction3] 


Q4. It is my job to make sure [STUDENT FIRST NAME] revises for upcoming tests. 


1 


2 


3 


Strongly Disagree 
Somewhat Disagree 
Somewhat Agree 
Strongly Agree 
Don’t Know 


Refused 
Terminated at this question 


[Role_construction4] 


Q5. It is important that | talk with [STUDENT FIRST NAME] about what [HE/SHE] is learning at 


school. 


Strongly Disagree 


Education Endowment Foundation 50 


Texting Parents 


2 Somewhat Disagree 
3 Somewhat Agree 
4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[ORDER Q6-Q11 inclusive randomized] 


[Invitation_1] 


Q6. I believe that [STUDENT FIRST NAME]’s school and teachers want me to help [HIM/HER] with 
[HIS/HER] homework each night. 


i Strongly Disagree 

2 Somewhat Disagree 
3 Somewhat Agree 

4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[Invitation_2] 


Q7. | believe that [STUDENT FIRST NAME]’s school and teachers want me to talk with [HIM/HER] 
about what [HE/SHE] is learning in school. 


1 Strongly Disagree 

2 Somewhat Disagree 
3 Somewhat Agree 

4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[Invitation_3] 


Q8. [STUDENT FIRST NAME]’s school and teachers believe | am a valuable partner in [HIS/HER] 
education. 
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1 Strongly Disagree 

2 Somewhat Disagree 
3 Somewhat Agree 

4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[Efficacy1] 


Q9. | have a lot of influence over how well [STUDENT NAME] performs in school. 


al Strongly Disagree 

2 Somewhat Disagree 
3 Somewhat Agree 

4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[Efficacy2] 


Q10. | have the ability to make sure [STUDENT FIRST NAME] is submitting all of [HIS/HER] 
homework. 


1 Strongly Disagree 

2 Somewhat Disagree 
3 Somewhat Agree 

4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[Efficacy3] 


Q11. | am confident that | can make sure [STUDENT FIRST NAME] is spending enough time 
studying. 
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1 Strongly Disagree 

2 Somewhat Disagree 
3 Somewhat Agree 

4 Strongly Agree 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


[behaviours] 


Now, we would like to learn about what kind of school-related information you discuss with 
[STUDENT FIRST NAME]. Please tell us if you have discussed any of the following with 
[STUDENT FIRST NAME] in the past month. [RANDOMIZE Q12-15 inclusive] 


[behavior_1] 
Q12. Have you asked [STUDENT NAME] what [HE/SHE] learned at school? 
0 No 
1 Yes 
90 Don’t Know 
91 Refused 


92 Terminated at this question 


[behavior 2] 
Q13. Have you talked with [STUDENT NAME] about [HIS/HER] attendance? 
0 No 
al Yes 
90 Don’t Know 
91 Refused 


92 Terminated at this question 


[behavior_3] 


Q14. Have you asked [STUDENT NAME] to revise for an upcoming test or exam? 
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91 


92 


No 

Yes 

Don’t Know 
Refused 


Terminated at this question 


Education Endowment Foundation 


Texting Parents 


54 


Texting Parents 


[behavior_4] 


Q15. Have you asked [STUDENT NAME] about [HIS/HER] grades? 


0 


a 


[experience] 


No 

Yes 

Don’t Know 
Refused 


Terminated at this question 


Q16. This school year, have you received any text messages with information about [STUDENT 
FIRST NAME]’s upcoming tests, what they have been learning about in school or any homework 
they have missed? 


0 


1 


2 


90 


91 
92 


No 

Yes 

Yes and provided comments about the messages (Capture response in Q16a) 
Don’t Know 


Refused 


Terminated at this question 


Q17. Imagine that next school year, [STUDENT FIRST NAME’S] school texted you information on 
upcoming tests, what is being taught that week and any homework [STUDENT FIRST NAME] 
might have missed. There would be an average of 2 texts per week. Would you be willing to pay 
for a text message service like that? 


0 


1 


No 

Yes 

Don’t Know 
Refused 


Terminated at this question 


TREATMENT ONLY, IF CONTROL, GO TO Q38 


[usefulness] 
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We are interested in making these text messages more useful for families and would like to hear 
more about how useful you found them. For each type of message listed, we would like to know 
how often you talked with [STUDENT NAME] about the content - never, rarely, sometimes, most 
of the time, always. You can also say if you did not receive a message like this. [RANDOMIZE 
order of Q18-21 inclusive] 


Q18. How often did you talk with [STUDENT NAME] about messages containing information 
about upcoming tests? Never, Rarely, Sometimes, Most of the Time, Always? Or did you not 
receive message like this? 


0 Never 

1 Rarely 

2 Sometimes 

3 Most of the time 

4 Always 

5 | did not receive this type of message 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


Q19. How often did you talk with [STUDENT NAME] about messages containing the grade 
received on an exam? Never, Rarely, Sometimes, Most of the Time, Always? Or did you not 
receive message like this? 


0 Never 

1 Rarely 

2 Sometimes 

3 Most of the time 

4 Always 

5 | did not receive this type of message 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


Q20. How often did you talk with [STUDENT NAME] about messages alerting you to homework 
that [STUDENT NAME] had not submitted? [Repeat options if requested] 


0 Never 


1 Rarely 
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Sometimes 

Most of the time 

Always 

| did not receive this type of message 
Don’t Know 


Refused 


Terminated at this question 


Q21. How often did you talk with [STUDENT NAME] about messages containing what the 
[STUDENT NAME] learned in school that day? [Repeat options if requested] 


0 


1 


2 


91 
92 


Never 

Rarely 

Sometimes 

Most of the time 

Always 

| did not receive this type of message [Go to QXX] 
Don’t Know 


Refused 


Terminated at this question 


Q22. Which two types of text that you received did you find most useful? Again, the types were 
upcoming tests, grades, missing homework, and what was taught that day? [Check all that apply] 


AE 


2 


3 


92 


Upcoming tests 

Grades 

Missing homework 

What was taught that day 
Don’t Know 


Refused 


Terminated at this question 


Q23. Which one of the messages that you received did you find least useful? [Repeat list if 


requested] 


1 


2 


3 


4 


Upcoming tests 
Test grades 
Missing homework 


What student learned 
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Don’t Know 


Refused 


Terminated at this question 


Q24. This year we tried to inform you about upcoming tests 4 days in advance and we'd like to 
know if you would have liked more or less notice. Ideally, how many days in advance would you 
want to know about an upcoming test for [STUDENT NAME]? Please give your answer in days. 
[Parent offers response] 


1 


2 


3 


91 
92 


1 day 

2 days 

3 days 

4 days 

5 days 

6 days 

7 days 

More than 7 days 

Other [Enter response in] 
Wouldn't want to know at all 
Don’t Know 


Refused 


Terminated at this question 


Q25. We tried to alert you about upcoming tests 4 days in advance and the day before the test. 
Which of the alerts were useful to you: 4 days in advance, 1 day in advance, both, or neither? 


1 


2 


3 


92 


4 days 

1 day 

Both 
Neither 
Don’t Know 


Refused 


Terminated at this question 


Q26. Now, thinking of all of the text messages together, how would you describe [STUDENT 
NAME]’s response when you brought up the information in the text messages? Was [his/her] 
reaction ‘Very negative, Somewhat negative, Neutral, Somewhat positive or Very positive? 


a 


Very negative 


Education Endowment Foundation 58 


Texting Parents 


2 Somewhat negative 
3 Neutral 

4 Somewhat positive 
5 Very positive 


90 Don’t Know 


91 Refused 
92 Terminated at this question 


Q27. [PreferredMode] Given the choice, please tell us if you would have liked to have received 
these messages in the following ways. By text? E-mail? Post? Facebook? School Gatewway 
(which is the school’s smartphone app)? A note sent home with your child? (Allow respondent 
to reply for each option, and SELECT ALL THAT APPLY) 


1 Text message (which is the way they were currently delivered) 
2 Email 

3 Regular postal service mail 

4 Facebook message 

5 Smartphone app 

6 Note sent home with student 

7 Other [Please record] 

90 Unsure 


91 Refused 


92 Terminated at this question 


Q28. What do you think about the frequency of messages sent? Were there too many, about the 
right amount or not enough messages? 


0 Too many 
1 About right 
2 Not enough 


90 Don’t Know 


91 Refused 


92 Terminated at this question 


Education Endowment Foundation 59 


Texting Parents 


Q29. If there were 3 pieces of information to share with you, would you prefer to receive 3 
separate texts at the same time or 1 long text? 


1 


2 


3 


90 


91 
92 


3 separate texts 

1 long text 

Other [Record in Q32a] 
Don’t Know 


Refused 


Terminated at this question 


I’m now going to ask what time you would have liked to receive each type of message. You would 
have received all messages at 4pm this year, but we’d really like to know if we could have sent 
them at a more convenient time. Please think about how you responded to the message and tell 
us whether it might have been better if we had sent them at different times. [RANDOMIZE Q30- 


32 inclusive] 


Q30. If you could choose, what time would you want to receive texts about upcoming tests? 
Please give your answer to the nearest hour. [Caller, please choose the appropriate option] 


1 


2 


91 


92 


Morning 

Between 12pm and 2pm inclusive 
3pm 

4pm 

5pm 

6pm 

7pm or later 

Don’t know 

Refused 


Terminated at this question 


Q31. If you could choose, what time would you want to receive missing homework messages? 
[Caller, please choose the appropriate option] 


1 


2 


3 


Morning 
Between 12pm and 2pm inclusive 


3pm 
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4 4pm 
5 5pm 
6 6pm 
7 7pm or later 


90 Don’t know 
91 Refused 


92 Terminated at this question 


Q32. What time would you want to receive conversation prompts on recent learning? [Caller, 
please choose the appropriate option] 


1 Morning 

2 Between 12pm and 2pm inclusive 
3 3pm 

4 4pm 

5 5pm 

6 6pm 

7 7pm or later 


90 Don’t know 
91 Refused 


92 Terminated at this question 
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Q33. Please tell us who you think should receive the messages? This can be more than one 
person and can include yourself, another parent or carer of your child, a relative, a family friend. 
[PLEASE CHECK ALL THAT APPLY] 


1 


2 


3 


Myself 

Another parent or carer of my child 
Another relative 

Family friend 

Don’t know 

Refused 


Terminated at this question 


Q34. If you have discussed the text messages with someone other than your child, please let us 
know who? The options are another parent or carer of your child, a relative, a family friend. 
[PLEASE CHECK ALL THAT APPLY] 


1 


2 


3 


Another parent or carer of my child 

Another relative 

Family friend 

| did not discuss the messages with anyone else 
Don’t know 

Refused 


Terminated at this question 


Thanks. There are just a few more questions to go, thanks so much. [RANDOMIZE Q36-38 inclusive] 


Q35. Were the messages about upcoming tests easy to understand? 


0 


1 
2 
3 


90 


91 


92 


No 


Yes 
Sometimes 
| didn’t receive this kind of text 


Don’t Know 
Refused 


Terminated at this question 
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Q36. Were the messages about missing homework easy to understand? 


0 No 

al Yes 

2 Sometimes 

3 | didn’t receive this kind of text 


90 Don’t Know 
91 Refused 


92 Terminated at this question 


Q37. Were the messages about recent learning easy to understand? 


0 No 

1 Yes 

2 Sometimes 

3 | didn’t receive this kind of text 


90 Don’t Know 
91 Refused 


92 Terminated at this question 
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Q38. Have you used School Gateway, the smartphone app that [SCHOOL NAME] uses, to find 
out more about [STUDENT NAME]’s learning? Please indicate all that apply from the following 
list: Upcoming school events, lunch, uniforms, negative behavior, good behavior, grades or 
anything else? (SELECT ALL THAT APPLY) 


0 No [Go straight to Q40] 

1 Upcoming school events 

2 Low lunch credit/free lunch information 
3 Uniform information 

4 Negative behaviour 

5 Good behaviour 

6 Grades 

7 Other [PLEASE STATE] 


90 Don’t know 


91 Refused 
92 Terminated at this question 


Q39. How often do you check the SchoolGateway app? Never, less than once a month, once a 
month, once a week, a few times a week, every day, or whenever you received a message? 


0 Never 

1 Less than once a month 

2 Once a month 

3 Once a week 

4 A few times a week 

5 Whenever | receive a message 


90 Don’t know 


91 Refused 


92 Terminated at this question 


[relationship] 
Q40. Thank you. What is your relationship to [STUDENT FIRST NAME]? [CALLER READS 
OPTIONS] 


a Parent 

2 Grandparent 
3 Aunt or uncle 
4 Step parent 
5 Sibling 

6 Cousin 
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92 


[language] 


Foster parent 
Other 
Refused 


Terminated 
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Q41. What is the primary language spoken in your household? (Answer is volunteered) 


1 


2 


92 


English 

Punjabi 

Urdu 

Bengali 

Polish 

Somali 

Gujarati 

Arabic 

Other [Please state] 


Don’t Know 
Refused 


Terminated at this question 


Q42. What do you consider to be your child’s ethnicity? 


OAN DOA BRWN PE 


PRPREPRPPPPRPPRPRB 
AONOAuORKRWNHERO 


White British 

Irish 

Traveller of Irish Heritage 
White other 

Gypsy/Romany 

Mixed White/Black Caribbean 
Mixed White/Black African 
Mixed White and Asian 

Any other mixed 

Indian 

Pakistani 

Bangladeshi 

Any Other Asian 

Black Caribbean 

Black African 

Any Other Black Background 
Chinese 

Any Other Ethnic Group 
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90 Don’t know 
91 Refused 


92 Terminated at this question 


[carer_education] 


Q43. Last question: what is the highest level of education you have completed?! (Answer is 
volunteered) 


Primary school abroad 

Secondary school abroad 

Secondary school in the UK 

Secondary school with some GCSEs/O Levels (or equivalent) 
Post-16 technical training/apprenticeship (or equivalent) 

A Levels (or equivalent) 

Undergraduate university degree 

Post-graduate university degree 


ONoOunBRWN BF 


90 Don't know 
91 Refused 


92 Terminated at this question 


Thank you so much for your time! Enjoy your [day/evening/night]! 
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Appendix 2: Memorandum of Understanding 


Parent Engagement Project 
Memorandum of Understanding 


This memorandum of understanding is between the school taking part in the research (named below) 
and the research teams located in Bristol University, Harvard University and the Centre for Effective 
Education in Queen’s University Belfast. 


Name of School: 


Its purpose is to clarify the roles and responsibilities of each organisation to facilitate the smooth running 
of the research project. 


To receive the Parent Engagement Project and take part in this study please read and sign this 
Memorandum of Understanding. 

Background 

The Centre for Effective Education at Queen’s University Belfast has been asked to carry out an 
independent evaluation of the implementation of the Parent Engagement Project in 34 schools. This 
research aims to find out what principals, teachers and pupils think of the programme, what were some 
of challenges and successes that were experienced and whether taking part in the programme has 
helped to improve children’s achievement in English, maths and science. 


The research team will ensure that parents are informed of the research and that opt-out consent is 
obtained for their child’s participation in the study. This study has been approved by the School of 
Education Research Ethics Committee in Queen’s University Belfast. 


Agreeing to take part in the research will mean the following: 
1. Random allocation of classes to get the programme 


Each participating school will put forward all the classes in Year 7, Year 9 (Key Stage 3) and 
Year 11 (Key Stage 4) to be included in the research. The CEE research team will randomly 
allocate either KS3 (Years 7 and 9) OR KS4 (Year 11) to get the programme. The Parent 
Engagement Project will be delivered between September 2014 and July 2015. 


The Key Stage Year Group that is not selected to get the programme will act as a comparison 
group (also called the control group). The parents in this group will not receive text messages 
about their child’s learning; instead they will receive ‘business as usual’. 


2. Facilitate ‘informed consent’ procedure 


Your school will be given a letter that must be distributed at the beginning of the 2014/15 school 
year to parents of pupils in Years 7, 9 and 11, whether or not they have been allocated to receive 
the programme. The letter will inform parents of the nature of the research and give them 2 
weeks to opt-out of the study. You will inform the research team of the pupils that have been 
withdrawn from the study. 


3. Facilitating the PEP programme 
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The text messages will either be sent automatically by Schoolcomms or manually by a Research 
Assistant (RA) who will have temporary and restricted access to your Schoolcomms account. 
Your school will not need to send any text messages, but for some texts you will need to either 
provide your RA with some information, or ensure certain information has been properly 
recorded on SIMS: 


Text Type 


Upcoming 
Assessment 


Description 


4 and 1 day warnings of 
upcoming tests & important 
deadlines 


Requirement from school 


Share assessment calendar 
with your RA, at least 1 in 
advance 


Text sent 
by 


Grades 


Missing 
Homework 


Grades for tests/important 
assignments 


Notification of missing 
homework 


Notify RA when grades on SIMS 
can be sent to parents 


Log behaviour report on SIMS 
as usual 


Automatic 


Automatic 


Conversation 
Prompt 


Summary of the day's lesson so 
parents can discuss 


Teachers must write prompts as 
timetabled 


Attendance 
Report 


4. Welcome event 


Summary of a student's 
attendance record for the term 


Share termly attendance data 
with research team 


You will send 2 representatives to attend a welcome event in September 2014 (exact date TBC). 
The 1-day event will explain the rationale for PEP and take you through the technical specifics 
of how the text messages will be sent in practice. We would recommend sending a Data 
Manager and a Departmental Head of English, Maths or Science. 


5. Liaising with your Research Assistant 


Your school will provide a primary contact that your assigned Research Assistant can call to 
discuss any issues/problems regarding the project. Your school will also participate in a half- 
termly conference call with your Research Assistant to discuss any issues regarding the 
research project. This conference call will ideally include a member of your SLT and the 
Departmental Heads of English, Maths and Science. 


6. Costs 


The costs of the text messages will be covered by the University of Bristol, as will other costs 
incurred as a result of participating in the project (Such as the printing of letters to be sent home, 


etc). 


7. Teacher online survey 
In Term Two (between January and April), teachers and administration staff involved in the 
Parent Engagement Project will each be asked to complete a short online survey to collect 
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information relating to the overall implementation of the programme and any challenges that 
were encountered. 


Principal, teacher and pupil interviews 


Your school might be one of three schools selected by the evaluation team to take part ina 
small number of interviews with principals and teachers and a focus group with young people 
to further explore overall perceptions of the programme and its effectiveness. If your school is 
selected, you will be contacted by the CEE research team at the start of Term 3 (after April) to 
discuss your participation. Conducting these interviews may require a short visit to the school 
by the research team; however the timing will be agreed with the school and teacher(s) to ensure 
disruption is minimized. 

Achievement Data Collection 

To explore the impact of the Parent Engagement Project on academic outcomes for all children 
in Years 7, 9 and 11 the CEE research team will collect achievement data in July 2015 using 
the following tests: 

Year 7 will be tested using a web based version of the Hodder Access Reading, Access 
Mathematics and Science Progress 

Year 9 will be tested using the Insight Maths, Science and Reading tests 

For Year 11 GCSE results will be used and these data will be collected by the evaluation 
team directly from the National Pupil Database 


The research team will pay for and coordinate the testing of Years 7 and 9 in close consultation 
with the school in order to minimize disruption. Schools will be provided with the data from the 
Year 7 and 9 tests. The research team will provide ongoing support to the school throughout 
the testing period. 


Addition pupil information 

The school will provide the CEE research team with the UPN for each child, name, date of birth 
and postcode. This information will enable the research team to link to the National Pupil 
Database records. 


Will participation in the study be kept confidential? 
Pupils’ test responses and any other pupil data will be treated with the strictest confidence. The 


respons 


es will be collected by the CEE research team. Named data will be matched with the National 


Pupil Database and shared with Bristol University, the Centre for Effective Education (the research 
team), the Education Endowment Foundation and the UK Data Archive for research purposes. No 
individual school or pupil will be identified in any report arising from the research. 


| understand that participation in this programme and evaluation is entirely voluntary and | am clear 
about my school’s role in this research study. | have been given the opportunity to ask questions and 


any con 


cerns that | may have raised have been addressed. 


Signed 


Date 


Position 
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Appendix 3: Parent Information Sheet and Consent Form 


Evaluation of the Parental Engagement Project 
Parent Information Sheet and Consent Form 


Hello, 


We are a research team from the Centre for Effective Education at Queen’s University Belfast, and we 
are working with a team from the University of Bristol to determine whether text messages are a good 
way for schools to communicate with parents about their child’s learning. We have been funded by the 
Education Endowment Foundation to carry out this research, and we would like you and your child to 
take part. Before you make a decision it is important for you to Know why the research is being done 
and what it will involve. If you do not want you or your child to take part, we will ask you to sign the 
attached consent form and return it to your child’s teacher. If you do wish to participate, you do not need 
to do anything, you will be automatically enrolled in the programme. 


Please take your time to read the following information carefully and discuss it with others if you wish. 


Why are we doing this study? 

Your child’s school is rolling out a new programme called the Parental Engagement Project, for the 
academic year beginning in September 2014. The programme involves different strategies for 
communicating with parents, via text message, about their child’s progress in school. The research aims 
to find out whether this is effective in improving young people’s performance at school. 


What will happen if my child takes part? 

If your child takes part, their class will be randomly chosen to either be part of the programme (meaning 
that you will receive text messages) or to continue as normal in which case you will not receive these 
additional text messages (only those that your school would normally send out). 

If your child’s class is not selected to be part of the programme then you will not receive these text 
messages but we would still like to monitor your child’s performance in school. 

The evaluation is being coordinated by the Queen’s Univeristy Belfast research team, and involves the 
following: 


Survey of parents: this short survey will take place in the Spring of 2015 and will ask for your 
feedback on receiving text messages from your child’s school. We will also ask for your consent 
to be contacted again for a short telephone interview to find out more about your experience, 
however you do not have to agree to this. 


Student academic achievement: in July 2015 students in Years 7 and 9 (both those who were 
part of the programme and those who were not) will be asked to take an English, maths and 
science test (as they usually would at this time of year). The tests will be delivered by the 
research team in your child’s school. For students in Year 11, we will access their GCSE results 
from the National Pupil Database. 

Focus groups with young people: we will ask a small number of young people who have been 
part of the programme to give us their views on the programme. This will take place in your 
child’s school. 


Students’ test scores will be shared with the school and all data will be treated with the strictest 
confidence. 
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Named data will be matched with the National Pupil Database and shared with Bristol University, the 
Centre for Effecive Education, the Education Endowment Foundation and the UK Data Archive for 
research purposes. We will not use your child’s name or the name of the school in any report arising 
from the research. 
This study has been approved by the Research Ethics Committee in the School of Education at Queen’s 
University Belfast. 


Does my child have to take part? 
You are free to agree for your child to take part in the evaluation of the programme. If you do NOT want 
your child to take part in the evaluation, please complete and return the form below. 


How long will the study last? 


This study will begin in September 2014 and will finish in August 2015. 


What if | change my mind? 

You can change your mind at any time during the study and decide that you do not want your child to 
take part anymore in either the focus group or the tests. You do not even have to give a reason why. 
Your decision to take part will in no way impact on your relationship with your child’s School, Queen’s 
University Belfast or Bristol University, now or in the future. 


Will our participation in the study be kept confidential? 


All information that is collected about you and your child during the course of the research will be kept 
strictly confidential. Research folders will be kept securely in a locked office at all times. Access to these 
folders will be restricted to study investigators and statisticians. Any information that is stored 
electronically will be kept securely on Queen’s University computers. 


How to contact us to find out more about the study 
If you would like to find out more about the study (even if you decide not to take part), please contact Dr 
Sarah Miller, at the address or phone number below: 


Dr Sarah Miller 

Centre for Effective Education 
School of Education 

Queen's University Belfast 
69-71 University Street 
Belfast 

BT7 1HL 


Telephone: 028 9097 5944 
E-mail: s.j.miller@qub.ac.uk 
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Evaluation of the Parent Engagement Project 
Parent Consent Form 


If you want to participate in this study, you do not need to do anything. 


If you DO NOT want to receive these text messages, or participate in the study, please complete this 
form and return it to <<INSERT SCHOOL ADMIN NAME>> by <<INSERT DATE 2 WEEKS AFTER 
ISSUE>>. 


Parent Name: 


Student Name: 
School Name: 
Please tick the boxes that apply. 
1. []1do not want to receive these text messages. 
2. []1!do not give consent for my child to take part in any focus groups 


3. []1!do not give consent for my child’s achievement data to be collected by the research team 


Signed Date 
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Table A4.1 Coefficients and standard errors for all models reported in Tables 8, 9 and 10 in the 
main report (English) 


English Addditi 
MI 
onal 
. . Mi addition 
Simple covariat . 
simple al 
model es 
model covariat 
(bootstr 
es 
ap) 
coefficie coefficie coefficie coefficie 
SE SE SE SE 
nt nt nt nt 
Intervention 0.034 0.015 -0.007 0.014 0.008 0.013 -0.026 0.014 
Pre-Reading score 0.616 0.007 0.616 0.006 0.641 0.006 0.641 0.006 
Key Stage Group -0.145 0.018 -0.150 0.014 
FSM eligibility -0.072 0.023 -0.077 0.020 
Ethnicity (white) 0.005 0.018 0.046 0.019 
School Size -0.008 0.007 0.009 0.024 
School Attainment 0.080 0.011 0.075 0.034 
School Pupil 
; 0.000 0.012 0.001 0.031 
Premium 
Constant 0.033 0.028 0.175 0.021 0.015 0.029 0.123 0.031 
School level 
— 0.019 0.005 0.113 0.008 0.145 0.021 0.118 0.017 
variation 
Pupil level 
—_ 0.490 0.006 0.697 0.007 0.718 0.005 0.714 0.005 
variation 


Education Endowment Foundation 


13 


Texting Parents 


Table A4.2 Coefficients and standard errors for all models reported in Tables 8, 9 and 10 in the 
main report (maths) 


Maths Addditi 
onal ml 
: : MI addition 
Simple covariat F 
simple al 
model es : 
model covariat 
(bootstr 
es 
ap) 
coefficie SE coefficie SE coefficie SE coefficie SE 
nt nt nt nt 
Intervention 0.067 0.013 0.029 0.014 0.067 0.012 0.039 0.013 
Pre-Maths score 0.753 0.006 0.751 0.006 0.759 0.006 0.759 0.006 
Key Stage Group -0.126 0.012 -0.128 0.014 
FSM eligibility -0.107 0.018 -0.115 0.017 
Ethnicity (white) -0.110 0.016 -0.110 0.017 
School Size 0.003 0.007 0.015 0.024 
School Attainment 0.025 0.009 0.028 0.034 
SceeLntpl -0.029 | 0.009 0.014 | 0.032 
Premium 
Constant 0.000 0.024 0.214 0.016 -0.008 0.024 0.194 0.030 
School level 
Bh 0.014 0.004 0.117 0.005 0.120 0.017 0.120 0.017 
variation 
Pupil level variation 0.345 0.005 0.583 0.006 0.604 0.005 0.599 0.005 
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Table A4.3 Coefficients and standard errors for all models reported in Tables 8, 9 and 10 in the 
main report (science) 


Science Addditio 
MI 
nal 
. . MI addition 
Simple covariat ; 
simple al 
model es 
model covariat 
(bootstr 
es 
ap) 
coefficie coefficie coefficie coefficie 
SE SE SE SE 
nt nt nt nt 
Intervention -0.013 0.018 0.017 0.021 -0.034 0.015 -0.026 0.015 
Pre-Maths score 0.582 0.008 0.579 0.007 0.631 0.007 0.625 0.007 
Key Stage Group 0.074 0.019 0.039 0.015 
FSM eligibility -0.116 0.024 -0.136 0.020 
Ethnicity (white) -0.107 0.019 -0.099 0.020 
School Size -0.020 0.009 0.000 0.046 
School Attainment 0.068 0.012 0.070 0.065 
School Pupil 
; -0.068 0.012 -0.044 0.060 
Premium 
Constant 0.045 0.050 0.077 0.024 0.013 0.049 0.086 0.050 
School level 
ay 0.069 0.019 0.243 0.009 0.256 0.034 0.236 0.032 
variation 
Pupil level 
iA 0.556 0.008 0.743 0.005 0.762 0.006 0.759 0.005 
variation 
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Table A4.4 Coefficients and standard errors for all models reported in Tables 8, 9 and 10 in the 
main report (absenteeism) 


Science Addditio 
MI 
nal 
. . MI addition 
Simple covariat : 
simple al 
model es 
model covariat 
(bootstr 
es 
ap) 
coefficie coefficie coefficie coefficie 
SE SE SE SE 
nt nt nt nt 
Intervention -0.013 0.018 0.017 0.021 -0.034 0.015 -0.026 0.015 
Pre-Maths score 0.582 0.008 0.579 0.007 0.631 0.007 0.625 0.007 
Key Stage Group 0.074 0.019 0.039 0.015 
FSM eligibility -0.116 0.024 -0.136 0.020 
Ethnicity (white) -0.107 0.019 -0.099 0.020 
School Size -0.020 0.009 0.000 0.046 
School 
; 0.068 0.012 0.070 0.065 
Attainment 
School Pupil 
; -0.068 0.012 -0.044 0.060 
Premium 
Constant 0.045 0.050 0.077 0.024 0.013 0.049 0.086 0.050 
School level 
i 0.069 0.019 0.243 0.009 0.256 0.034 0.236 0.032 
variation 
Pupil level 
3%, 0.556 0.008 0.743 0.005 0.762 0.006 0.759 0.005 
variation 


Education Endowment Foundation 76 


Texting Parents 


Appendix 5: Number of texts sent 


Table A5 Number of texts sent for each subject per school 


School English Maths Science Total 
ad; 11 17.9 11.7 40.6 
2 25.5 25.6 26.2 77.3 
3 12.1 16.6 10.2 38.9 
4 16.3 18.5 12 46.8 
5 12.2 15.4 14.9 42.5 
6 6.5 10.3 3.7 20.5 
7 15.6 16.3 25 56.9 
8 3 9 3 15.0 
9 27.3 21.8 12 61.1 
10 10.3 12.4 10.7 33.4 
11 18 21.3 22.8 62.1 
12 13.5 9 6 28.5 
13 20 9.3 18.5 47.8 
14 18.3 27.8 22.1 68.2 
15 25.6 4.8 13.2 43.6 
16 18.8 18.1 33.9 70.8 
17 20.7 13.4 12.8 46.9 
18 26.7 15.4 26.6 68.7 
19 23.3 27.6 22.3 73.2 
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20 23.7 30.8 18.4 72.9 
21 13.9 29.9 26.7 70.5 
22 19.8 16.9 13.7 50.4 
23 11.2 22.3 30.3 63.8 
24 13 15.9 9 37.9 
25 14.2 6.9 9 30.1 
26 19 25.7 23.4 68.1 
27 9.9 20.6 10.4 40.9 
28 7.2 12.5 fd 27.0 
29 14.4 15 L3 36.7 
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Appendix 6: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. Cost ratings are awarded using the following criteria. 


Cost Description 
£ Very low: less than £80 per pupil per year. 
£E Low: up to about £200 per pupil per year. 


ELE Moderate: up to about £700 per pupil per year. 
ELLE High: up to £1,200 per pupil per year. 
ELLEEE Very high: over £1,200 per pupil per year. 
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Appendix 7: Padlock rating 


7 July 2016 Complete by Anneka Dawson 


2. Power: 5. Threats to 
1.Design: ee 3. Attrition: 4. Balance: SaaS, Pa 
h ae ee idity: 
What is the ae What is the Adjustment a 


minimum Adjustment to 
quality of the (V-) Me) Mel ge) «) to rating to 
fol} ¢-eie-]e) (= account for issues 
design of the out from the account for 
uation? effect at the iatienD balance? with 
evaluation? valuation: ance? 
: start? bette - m interpretation? 
Evaluation design Implementation Analysis and interpretation 


Rating 1. Design 2. Power 3. Attrition 4. Balance LoYa Bal aysleom ce) 
(Le) =59) validity 


Well-balanced on 
I) 
Fair and clear experimental 
design (RCT, RDD) : 
Well-matched comparison 
(quasi-experiment) : < 30% 


Matched comparison 

(quasi-experiment) . < 40% 

Comparison group with 

poor or no matching . < 50% 

No comparator 0 Imbalanced on re 


The final security rating for this trialis 3 @. 
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Appendix 8: Post-publication analysis 


This appendix was added to the published report in June 2017 


The evaluation team was asked to consider whether all sources of variation were appropriately 
accounted for in the two-level model that was originally used to analyse the data from the PEP evaluation 
(see the ‘analysis’ section of the main report). 


A summary of the findings of this original (complete case) analysis are reported in Table A8.1, below 
(taken from Table 8 of the main report). As can be seen, the intervention results in a small but statistically 
significant improvement in maths and absenteeism and no improvement in science attainment. These 
three results were supported by further sensitivity analyses i.e. the inclusion of additional covariates, 
bootstrapping and multiple imputation (to account for the large proportion of missing pre-test (KS2) data) 
— see the full report for further details. 


Whilst the original primary analysis also indicates that the intervention improves English attainment, this 
was not supported by subsequent sensitivity analyses and so the evidence for this outcome is much 
less certain or consistent than the evidence relating to maths, absenteeism and science. 


Table A8.1: Original primary analysis using a two-level model and two covariates: group allocation and 
pre-test score 


Effect size sig 
“English —~—~«<0.033.—~=*=—=<“<«‘CWDH 
Maths 0.067 <0.001 
Science -0.013 0.467 
Absence -0.054 <0.001 


Further analysis using a three level model 


The data were further analysed using a three level (random intercepts) model with pupils (level 1) nested 
within year groups (level 2) nested within schools (level 3). Year group rather than key stage group was 
used as the level two cluster variable to take account of any clustering effects associated with 
differences between years 7 and 9 (Key Stage 3). As per the original models, the new models included 
the relevant post-test score as the dependent variable and just two variables as independent variables: 
group membership (intervention or control) and the relevant pre-test score. 


As can be seen from Table A8.2, the positive effects for maths and absenteeism remain relatively 
unchanged and whilst the p values have increased, the results for these two outcomes remain 
statistically significant. Science, as per the original analysis, remains not statistically significant. The 
positive effect for English however has all but disappeared. Given that the evidence for the effect of the 
intervention on this outcome was identified as weak originally, this finding is perhaps not surprising. 


Education Endowment Foundation 81 


Texting Parents 


Table A8.2: Primary analysis using a three-level model and two covariates: group allocation and pre- 
test score 


Effect size 95% Cl sig 
English =—-0.015.—S—Ss«*-.061, 091] 0.697 
Maths 0.067 [.006, .128] 0.032 
Science -0.051 [-.243, .141] 0.605 
Absence -0.050 [-.084, -.015] 0.005 


Multiple imputation 


All four new models were also estimated using multiply imputed data (Table A8.3). This sensitivity 
analysis confirms - as per Table 2 - the positive effect of the intervention on maths and absenteeism 
and no effect in relation to English and science. 


Table A8.3: Primary analysis using a three-level model on multiply imputed data 


Effect size sig 
English 0.004 0.920 
Maths 0.070 0.026 
Science 0.026 0.767 
Absence -0.049 0.006 


Calculation of effect sizes 


The effect sizes reported in Tables A8.1, A8.2 and A8.3 were calculated using a conservative estimate 
of group variance, based on the pre-test scores of the sample for each outcome. Using instead the total 
variance from the model to calculate effect sizes results in a slight inflation of effects, see Table 4. 


Table 4: Effect sizes arising from using two alternative sources of variance in the effect size calculation 


Effect size Effect size 
using SD of using total variance 
pre-test scores from model 
English 0.015 0.021 
Maths 0.067 0.111 
Science -0.051 -0.065 
Absence -0.050 -0.066 


Conclusion 
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Given all of the above, it is our conclusion that taking into account the additional source of variation at 
the year group level does not substantially change the main conclusions of the original analysis. 
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